summaryrefslogtreecommitdiffstats
path: root/mirrors
AgeCommit message (Collapse)AuthorFilesLines
2013-01-14Add migration to move country data down to the URL levelDan McGee1-0/+74
Rather than have the weird indirection we need now to find the right country for URLs, just always store it on the URL. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Use content_type and not mimetype on HttpResponse()Dan McGee1-2/+2
Bug #16519 in Django deprecates mimetype, so update our code accordingly. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-12Round two of mirror status query improvementsDan McGee1-9/+10
This seems to generate much more performant queries at the database level than what we were previously doing, and also doesn't show duplicate rows. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-12-27Ensure mirror protocols are distinctDan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-21Fix mirror URL duplication in status viewDan McGee1-1/+2
We need to ensure we don't duplicate URLs in the status view, so add a distinct() call back in to the queryset when it was inadvertently dropped in commit a2cfa7edbb. This negates a lot of the performance gains we had, unfortunately, so it looks like a nested subquery might be more efficient. Disappointing the planner can't do this for us. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-16Use Python 2.7 dictionary comprehension syntaxDan McGee1-7/+4
Rather than the old idiom of dict((k, v) for <> in <>). Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-16Optimize mirror status data fetchingDan McGee1-8/+16
Now that we have as many mirror URLs as we do, we can do a better job fetching and aggregating this data. The prior method resulted in a rather unwieldy query being pushed down to the database with a horrendously long GROUP BY clause. Instead of trying to group by everything at once so we can retrieve mirror URL info at the same time, separate the two queries- one for getting URL performance data, one for the qualitative data. The impetus behind fixing this is the PostgreSQL slow query log in production; this currently shows up the most of any queries we run in the system. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-13Prefetch the available protocols on the mirror overview pageDan McGee2-5/+8
Otherwise we are doing one query per mirror, which at this point is over 100 separate queries. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-12Add get_latest_by to MirrorLog Meta classDan McGee1-0/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Add title to mirror status data pointsDan McGee1-1/+3
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Mirror graph tweaking after usage with real dataDan McGee3-4/+6
* Clamp y-axis minimum to 0. * Don't plot `is_success == false` values. * Ensure URLs are sorted predictably. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Re-add nice() calls for mirror status axesDan McGee1-2/+2
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-11Mirror status graph, now with points AND linesDan McGee2-17/+44
We might have to tweak the interpolation method once we see this with real data, but for now it looks really pretty locally. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Enable mirror status graph resizingDan McGee1-15/+28
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Add initial version of mirror status chartDan McGee1-0/+97
Still have some hardcoded stuff to rip out and replace to make this a bit more dynamic on things like sizing, but for now, this is a great start. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Allow filtering retrieved mirror statuses by mirror_idDan McGee2-8/+19
When we don't need them all, no need to fetch them all. Let the database do the work for us, hopefully. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-10Add mirror extended status JSON viewDan McGee2-3/+35
When asking for status for a single mirror, we can include logs from the past 24 hours in addition to the normal information we provide. This is slated for usage by a frontend graph still to come, similar to those on the NTP pool website. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-10-11Make mirror log time query a bit more efficientDan McGee1-4/+6
We don't need the full mirror log objects; we just need a very small subset of values from them here to do the required math and object building. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-09-19Sort mirrorlist by English country name, not ISO codeDan McGee1-1/+1
Fixes FS#31503. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-08-08Push more default values down into the databaseDan McGee4-5/+5
This makes it easier to do manual manipulation/insertion/etc. at the database level, as well as just making things act more sane from an overall software stack perspective. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-08-01Declare 'enums' at class scopeDan McGee2-10/+9
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-25Replace deprecated direct_to_template() with render() shortcutDan McGee1-10/+9
Now that Django actually provides a concise way to use a RequestContext object without instantiating it, we can use that rather than the old function-based generic view that worked well to do the same. Additionally, these function-based generic views will be gone in Django 1.5, so might as well make the move now. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-25Remove custom utc_now() function, use django.utils.timezone.now()Dan McGee2-11/+13
This was around from the time when we handled timezones sanely and Django did not; now that we are on 1.4 we no longer need our own code to handle this. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-09Correctly reassign queryset with added annotation in mirror statusDan McGee1-1/+1
This was a dumb oversight on my part in commit 0f3c894e7a0. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-09mirrorcheck: Don't use bulk_create on sqlite3Dan McGee1-2/+6
It isn't worth it, as we run into the 999 max SQL statement variables issue when using it on any significant amount of mirrors. Since this is just a development database setup, and it isn't a command we need to run especially fast, we can ditch it. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-09Don't include StdDev on sqlite3 mirror status queryDan McGee1-3/+9
Because this function isn't shipped by default, it makes more sense to just omit it completely from the query we do to build the tables on this page when in development. Substitute 0.0 for the value so the rest of the calculations and display work as expected. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-09Handle HTTPException being thrown in mirrorcheckDan McGee1-0/+6
Managed to see this bubble up today when running the mirrorcheck command on a less than ideal connection that was experiencing timeouts at the wrong time. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-19mirrors: add an alternate_email columnDan McGee3-2/+71
We have a lot of these in the freeform text area in the mirror notes; attempt to make this data usable as necessary if we want to do some sort of mirror notification automation in the future. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-14Add ability to restrict status report to single tierDan McGee3-4/+18
This should make it easier to catch errors in our Tier 1 mirrors. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-14Add rsync support to mirrorcheck and other small improvementsDan McGee1-41/+89
The main changes in this patch implement rsync:// protocol checking support by calling the rsync binary, requested in FS#29878. We track and log much of the same things as we already do for FTP and HTTP URLs- check time, last sync, total check duration, etc. Also added in this patch is a configurable timeout value which defaults to the previous hardcoded value of 10 seconds; this can be passed as an option to the mirrorcheck command. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-14Change mirror log error text to unlimited lengthDan McGee2-1/+71
Use TextField rather than a limited-length CharField; leave it up to the code filling this field to determine an appropriate length. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-14Don't limit protocols returned by mirror status functionDan McGee1-2/+0
If results weren't available for certain URLs, they won't show up anyway in this list, and if we start to check rsync URLs, then we want their values to come back in this status list. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-12Use python json module directly in place of simplejsonDan McGee1-3/+2
As of Python 2.6, this is a builtin module that has all the same functions and capabilities of the Django simplejson module. Additionally simplejson is deprecated in the upcoming Django 1.5 release. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-02Add a smart protocol filter to mirrorlist generatorDan McGee1-15/+40
This will only list FTP mirrors for a given country if there are no HTTP mirrors available, since FTP must die. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-02Allow mirrorlist generator pattern to match any protocolDan McGee2-9/+14
Add a helper method that checks if we know about the protocol; if so, we can spit out a URL for it. This allows (if you are insane) generation of an rsync mirrorlist, for instance. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-30Use a custom User-Agent when checking mirror URLsDan McGee1-7/+10
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-29mirrorresolv: only run update query if values changedDan McGee1-1/+7
98% of the time, we won't need to update the existing values as it will be the same as the prior run of this command. Do a quick check of the old and new values and don't send anything to the database if there is no need for an update. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-26Remove now-unused NullCharFieldDan McGee9-18/+11
It would be straightforward if we just needed to remove the field class definition, but we also need to update all migrations that referenced it so we don't have problems starting and migrating from scratch. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Ensure sorted order of mirrors in status page matches with JSDan McGee2-8/+7
We had one sorting order in the backend, and another once the JS sorting routine kicked in. Match them so we aren't doing more on the client-side on initial display than we have to. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Clean up mirrorlist generation form a bitDan McGee1-3/+15
Rename template since it really isn't an index of mirrors at all, and convert the form to use multiple checkboxes for both ftp/http and ipv4/ipv6 selection. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Finish django countries implementationDan McGee4-34/+130
* Add a migration to drop the old countries field. * Update all templates/views/utility methods to point at the new country field and dereference it as necessary. * Add the flags images to a few views where it makes sense. * Cleanup the download page layout quite a bit. * Bump the mirror status JSON version to 3; add country_code attribute. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Add django_countries country code fields and population migrationsDan McGee4-3/+173
This adds these columns and attempts to populate them with data from our existing country column data. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-25Rename mirror country fields to country_old in prep for normalizationDan McGee5-14/+82
We're going to move to using ISO 2 character codes via django countries, so start by moving the old data out of the way first. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-20Fix ContentType related migrationsDan McGee1-1/+20
This should fix the issues reported in FS#23228 for the most part. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-09Don't blow up when no mirror status data is availableDan McGee1-1/+1
The check here was wrong before; in the case of no mirror log entries the returned value will not be empty, but will contain two empty values. Check the values instead to see if we have valid data available. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-08Add migrations for on_delete attributesDan McGee1-0/+68
I don't think these end up doing anything at the database level, but since South insists on generating alter_column() calls for these changes, do them once so they don't keep reappearing in auto-generated future migrations. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-07Choose an up-to-date mirror for download URLsDan McGee1-0/+34
Given that we collect a lot of mirror status data, we can utilize it to ensure the download link on the website actually works and newly-added packages have actually been mirrored out. Add a method that attempts to use the mirror status data to determine a mirror we should redirect our download requests to. This can change on a regular basis, and falls back to the old method if no mirror status data is available. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-26Bump signoffs and mirror status JSON API versionsDan McGee1-1/+1
Due to datetime formatting changes in Django 1.4, we know follow the ECMA specification more closely and use 'yyyy-mm-ddThh:mm_ssZ' format. As this could break existing users of the JSON data, bump the version. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24Use bulk_create() when inserting mirror log entriesDan McGee1-2/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24Make all datetime objects fully timezone awareDan McGee2-7/+11
This is most of the transition to Django 1.4 `USE_TZ = True`. We need to ensure we don't mix aware and non-aware datetime objects when dealing with datetimes in the code. Add a utc_now() helper method that we can use most places, and ensure there is always a timezone attached when necessary. Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-24-g4f1b (git 2.32.0) at 2024-11-22 02:53:59 +0100