summaryrefslogtreecommitdiffstats
path: root/mirrors/management
AgeCommit message (Collapse)AuthorFilesLines
2013-11-07Django 1.6 upgrade, deprecation cleanupDan McGee1-1/+1
PendingDeprecationWarning: commit_on_success is deprecated in favor of atomic. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-01Guard logging operation to prevent needless text joinDan McGee1-2/+3
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-06-01Honor mirror URL active attribute in several placesDan McGee2-2/+2
Signed-off-by: Dan McGee <dan@archlinux.org>
2013-04-17Various minor code cleanups and fixesDan McGee1-9/+3
Most of these were suggested by PyCharm, and include everything from little syntax issues and other bad smells to dead or bad code. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-13mirrorcheck: s/if/elif/ when determining what check function to runDan McGee1-1/+1
This was a silly thinko here; it caused the logs to fill up with a bunch of 'unknown url type: rsync' errors. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-13Don't check FTP + IPv6 combinationDan McGee1-1/+15
Very few, if any, FTP servers support connections over IPv6. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Don't add blank options to rsync command lineDan McGee1-1/+5
Rsync doesn't like this so much: Unexpected remote arg: rsync://mirror.example.com/archlinux/lastsync rsync error: syntax or usage error (code 1) at main.c(1214) [sender=3.0.9] Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Teach mirrorcheck management command about check locationsDan McGee1-17/+46
This adds the -l/--location argument to the command in order to pass in a check location that we are currently running from. This locks the IP address family to the one derived from the address on that location, and stores any check results tagged with a location ID. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-03-07Revert "mirrorcheck: Don't use bulk_create on sqlite3"Dan McGee1-6/+1
This reverts commit 3c4ceb16. We don't need this anymore as bulk_create gets automatic batching now on sqlite3 so it is safe to use. Signed-off-by: Dan McGee <dan@archlinux.org>
2013-02-09Use 'update_fields' model.save() kwargDan McGee1-1/+1
This was added in Django 1.5 and allows saving only a subset of a model's fields. It makes sense in a few cases to utilize it. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-25Remove custom utc_now() function, use django.utils.timezone.now()Dan McGee1-3/+4
This was around from the time when we handled timezones sanely and Django did not; now that we are on 1.4 we no longer need our own code to handle this. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-09mirrorcheck: Don't use bulk_create on sqlite3Dan McGee1-2/+6
It isn't worth it, as we run into the 999 max SQL statement variables issue when using it on any significant amount of mirrors. Since this is just a development database setup, and it isn't a command we need to run especially fast, we can ditch it. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-09Handle HTTPException being thrown in mirrorcheckDan McGee1-0/+6
Managed to see this bubble up today when running the mirrorcheck command on a less than ideal connection that was experiencing timeouts at the wrong time. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-14Add rsync support to mirrorcheck and other small improvementsDan McGee1-41/+89
The main changes in this patch implement rsync:// protocol checking support by calling the rsync binary, requested in FS#29878. We track and log much of the same things as we already do for FTP and HTTP URLs- check time, last sync, total check duration, etc. Also added in this patch is a configurable timeout value which defaults to the previous hardcoded value of 10 seconds; this can be passed as an option to the mirrorcheck command. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-30Use a custom User-Agent when checking mirror URLsDan McGee1-7/+10
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-29mirrorresolv: only run update query if values changedDan McGee1-1/+7
98% of the time, we won't need to update the existing values as it will be the same as the prior run of this command. Do a quick check of the old and new values and don't send anything to the database if there is no need for an update. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24Use bulk_create() when inserting mirror log entriesDan McGee1-2/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24Make all datetime objects fully timezone awareDan McGee1-1/+4
This is most of the transition to Django 1.4 `USE_TZ = True`. We need to ensure we don't mix aware and non-aware datetime objects when dealing with datetimes in the code. Add a utc_now() helper method that we can use most places, and ensure there is always a timezone attached when necessary. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-16Small Python convention fixupDan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-09Management command cleanupDan McGee1-5/+0
Now that we aren't seeing odd segfaults and hung tasks, we can remove the traceback stuff from the scripts. Also use the 'io' module only, it has been long enough. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-18mirrors: pylint discovered cleanupsDan McGee2-4/+4
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-18Never parse generated lastsync fileDan McGee1-22/+2
We should be encouraging our mirrors to serve us the original file, not something they create and come up with. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-27Auto-resolve mirror URLs on saveDan McGee1-6/+3
This prevents people from having to mess with these checkboxes at all in the admin, and we incur no delay on their initial values being correct waiting for the cron job to run. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-22Slight tweaks to mirror commandsDan McGee2-10/+9
We know we are doing updates when setting IPv4/IPv6 information, so set force_update to True to save the useless select query on each save(). For mirror checks, use a less cumbersome deque for the results since it is also thread-safe, and have all the log entries committed in one go. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-31Defer format string substitution to loggerDan McGee2-8/+10
Don't use 'fmtstr % (arg1, arg2)' type format; logger can be passed a format string and the arguments to populate it. Saves a bit of work for strings that never end up getting displayed anyway. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13Move database interaction out of the threaded sectionDan McGee1-14/+22
We were seeing a lot of hangs and long-running never-ending processes. This might be due to some multithreading issues within Django, so move the save() calls to a loop after the join() on the threads doing the mirror polling. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-14Mirror check script cleanupDan McGee1-10/+7
Don't import all the constants from logging, just use logging.* instead. Also, fix some typos that somehow snuck into one of my commits. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-14Add mirrorresolv manage.py commandPyroPeter1-0/+55
Dan: * Fix up some style issues such as spacing between operators * Ensure one failed lookup doesn't crash the whole script * Be silent out of the box if there are no errors, just like mirrorcheck Signed-off-by: PyroPeter <abi1789@googlemail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-12mirrorcheck: catch and handle another socket timeout caseDan McGee1-3/+10
We were seeing processes hang on the Arch server. It looks like there are ways for socket.timeout to come out of the main check code, so add another except block to catch this case. In addition, make sure we always call task_done() even on failures so processes eventually die. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-10-07mirrorcheck: record duration on file not found errorsDan McGee1-1/+16
On an HTTP 404, FTP 550, or inability to parse the lastsync file, record the duration of the check even though we couldn't get a time from the mirror. This allows for these checks to show up as completed but in error, which is more what. Previously, inability to parse the date was also recorded as a success, so change that to be a failure and record an error message with it. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-22Small updates to mirrorcheck commandDan McGee1-3/+5
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Add debugger when having to parse lastsync fileDan McGee1-0/+2
Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-21Add 'mirrorcheck' commandDan McGee3-0/+149
This does the actual work of going out and checking the mirror status. In short, it polls every active mirror URL for the 'lastsync' file and then records the appropriate details. These include the contents of that file, how long the total time to retrieve took, and any errors encountered. In order to finish up a bit faster, we spawn several threads to do the actual work. This parallelization allows the whole check process to take around 30 seconds rather than several minutes. Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-24-g4f1b (git 2.32.0) at 2024-11-22 11:09:16 +0100