summaryrefslogtreecommitdiffstats
path: root/mirrors
AgeCommit message (Collapse)AuthorFilesLines
2013-12-15Use stable parameters for cacheable functionDan McGee1-2/+3
It doesn't do much good to mark a function as cacheable if we call it every single time with different arguments due to using the current date and time. Fix it by passing the offset in instead. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-12-14Add update query for extracting Flyspray bug numberDan McGee1-0/+8
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-12-14Add Flyspray Bug field to mirror modelDan McGee3-1/+86
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-12-14Show all mirror status data to authorized usersDan McGee2-14/+24
Regardless of whether the mirror URL is active or not, we often have data we can show the end user, especially if mirror admins care to see the data we've been gathering. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-11-27Stop using Django-provided floatformat template tagDan McGee1-3/+8
It turns out this is a HUGE part of our slow mirror status template rendering, due to the internal workings. Everything is converted to a Python decimal object which is way slower than just staying in native floating point. Given we are always dealing with floats when we need to do our formatting, a home-rolled template tag can accomplish this much faster. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-11-07Django 1.6 upgrade, deprecation cleanupDan McGee1-1/+1
PendingDeprecationWarning: commit_on_success is deprecated in favor of atomic. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-11-07Django 1.6 upgrade, deprecation cleanupDan McGee1-0/+6
PendingDeprecationWarning: Creating a ModelForm without either the 'fields' attribute or the 'exclude' attribute is deprecated - form needs updating Signed-off-by: Dan McGee <dan@archlinux.org>
2013-07-13Fix completion percentage calculation in mirror statusDan McGee1-2/+2
We sometimes record a duration even on a failed fetch attempt, such as if we get an HTTP 404. However, we never record a last_sync value on a failed fetch. Use this field instead to sum up the total number of successful checks. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-20Re-enable caching for somewhat expensive mirror status queryDan McGee1-1/+1
This should be a small enough chunk of data that it isn't super expensive to put into and pull out of memcached. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-01Guard logging operation to prevent needless text joinDan McGee1-2/+3
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-01Ensure only active mirror URLs are listedDan McGee2-1/+2
This fixes things up on the download page as well as the individual mirror details page. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-01Whoops, committed a little bit too muchDan McGee1-3/+0
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-01Honor mirror URL active attribute in several placesDan McGee4-7/+14
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-01Add active column to mirror URLsDan McGee2-0/+84
2013-05-22Begin removal of FTP mirror supportDan McGee5-35/+10
FTP is a terrible protocol these days compared to HTTP. IPv6 support is spotty at best, it is much slower for the connect/begin transfer cycle, and overall just doesn't provide anything HTTP does better. Start killing bits that we've added to treat FTP as a first-class protocol and regulate it to the back seat. The expectation here is once this commit goes live to the production site, the FTP mirror URLs themselves will get removed completely from the database, and the FTP protocol object itself will get deleted. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-05-11Add mirror error logs to mirror details pageDan McGee1-2/+9
Give a window of 7 days for logs here rather than the default 24 hours we do on the main status page since we are only retrieving details for a single mirror with a handful of URLs. This should make it easier to have all information regarding one mirror in a single location. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-20Fix some None issues with sqlite3 and mirror statusDan McGee1-2/+4
If certain attributes came back from the database as NULL, we had issues parsing them. Pass None/NULL straight through rather than trying to type-convert. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-17Various minor code cleanups and fixesDan McGee3-13/+7
Most of these were suggested by PyCharm, and include everything from little syntax issues and other bad smells to dead or bad code. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-17Remove COUNTRY_LOOKUP global variableDan McGee1-3/+3
This is only used in one place, so it makes more sense for it to not be globally accessible. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-14Tweaks to mirror status chart generationDan McGee1-10/+12
* Use 'jQuery' rather than '$' * Use same colors for URLs in every chart for clarity Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-14Draw one mirror status graph per check locationDan McGee2-20/+51
Rather than lump it all together and have odd spikes depending on which side of the Atlantic checked a mirror in a given timeslot, draw a chart per check location. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-14Add a JSON view for retrieving mirror check locationsDan McGee2-2/+32
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-14Remove cache_function decorator from a few spotsDan McGee1-2/+0
The benefit of these storage operations might be outweighed by the cost, especially given how infrequently these functions are called. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-14Reduce mirror status query madnessDan McGee1-55/+91
Move completely to custom SQL for this logic. The Django ORM just doesn't play nice with the kind of query we are looking to do, so it is easier to do using raw SQL. The biggest pain factor here is in supporting sqlite as it doesn't have nearly the capabilities in handling datetime types directly in the database, as well as having some different type conversion necessities. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-13Support only a single mirror ID in error/status retrievalDan McGee2-11/+11
This simplifies things and makes injecting this single mirror ID into custom SQL a whole lot easier. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-13Calculate average URL delay in the databaseDan McGee1-19/+30
Rather than doing this in the Python code and needing 12,000+ rows returned from the database, we can do it in the database and get fewer than 300 rows back. If I recall correctly, the reason this was not done originally was due to our usage of MySQL and some really bad date math/overflow stuff it did when the interval between last_sync and check_time were greater than about a week. Luckily, we have switched to using a more sane database. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-11Allow mirror rsync IPs to be IPv4/IPv6 addresses or networksDan McGee4-15/+136
This gives us a bunch more flexibility on this field, and now supports all the options that the rsync config file supports. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-29Add IP family lookup to CheckLocation modelDan McGee1-0/+9
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-13mirrorcheck: s/if/elif/ when determining what check function to runDan McGee1-1/+1
This was a silly thinko here; it caused the logs to fill up with a bunch of 'unknown url type: rsync' errors. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-13Don't check FTP + IPv6 combinationDan McGee1-1/+15
Very few, if any, FTP servers support connections over IPv6. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Don't add blank options to rsync command lineDan McGee1-1/+5
Rsync doesn't like this so much: Unexpected remote arg: rsync://mirror.example.com/archlinux/lastsync rsync error: syntax or usage error (code 1) at main.c(1214) [sender=3.0.9] Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Teach mirrorcheck management command about check locationsDan McGee1-17/+46
This adds the -l/--location argument to the command in order to pass in a check location that we are currently running from. This locks the IP address family to the one derived from the address on that location, and stores any check results tagged with a location ID. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Add location ID to mirror logsDan McGee2-0/+84
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Add family property to mirror check locationDan McGee1-0/+7
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Revert "mirrorcheck: Don't use bulk_create on sqlite3"Dan McGee1-6/+1
This reverts commit 3c4ceb16. We don't need this anymore as bulk_create gets automatic batching now on sqlite3 so it is safe to use. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Add mirror CheckLocationAdminDan McGee1-1/+17
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Add 'created' field to more mirror modelsDan McGee2-2/+104
We have been better about doing this to most of our models, but the ones here didn't have a created field. Add it where appropriate and set a reasonably old default value. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Add mirror check locations modelDan McGee2-1/+105
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-05Add HTTPS mirror protocol fixtureDan McGee1-2/+11
For now, it is not included in the default selection, but we have a few existing mirrors that do support it. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-02-09Use 'update_fields' model.save() kwargDan McGee1-1/+1
This was added in Django 1.5 and allows saving only a subset of a model's fields. It makes sense in a few cases to utilize it. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-20Lengthen the mirror rsync IP address fieldDan McGee2-2/+69
Make it long enough to support a full-form IPv6 address with a subnet. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-16Remove debugging print statementDan McGee1-1/+0
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Support mirror status JSON by tierDan McGee2-1/+8
Just as we do for the normal status HTML view. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Ensure URLs without check data work on mirror details pageDan McGee1-7/+11
Less noticeable in production as the templates don't show '@@@INVALID@@@' there, but we were trying to access attributes that don't actually exist on certain mirror objects. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Drop country column from mirror tableDan McGee5-22/+81
We now always look for this information at the URL level, not the mirror level. This simplifies quite a bit of code in and around the mirror views. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Add migration to move country data down to the URL levelDan McGee1-0/+74
Rather than have the weird indirection we need now to find the right country for URLs, just always store it on the URL. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-14Use content_type and not mimetype on HttpResponse()Dan McGee1-2/+2
Bug #16519 in Django deprecates mimetype, so update our code accordingly. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-01-12Round two of mirror status query improvementsDan McGee1-9/+10
This seems to generate much more performant queries at the database level than what we were previously doing, and also doesn't show duplicate rows. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-12-27Ensure mirror protocols are distinctDan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-11-21Fix mirror URL duplication in status viewDan McGee1-1/+2
We need to ensure we don't duplicate URLs in the status view, so add a distinct() call back in to the queryset when it was inadvertently dropped in commit a2cfa7edbb. This negates a lot of the performance gains we had, unfortunately, so it looks like a nested subquery might be more efficient. Disappointing the planner can't do this for us. Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-24-g4f1b (git 2.32.0) at 2024-11-22 00:06:06 +0100