summaryrefslogtreecommitdiffstats
path: root/devel/management
AgeCommit message (Collapse)AuthorFilesLines
2012-07-09Work around bulk_create limitations in sqlite3 in reporeadDan McGee1-6/+28
Given the 999 SQL statement variable limit, we can easily hit it when updating a package with thousands of files or a few hundred depends. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-09reporead: disable FULL synchronous writes for sqlite3Dan McGee1-0/+6
At least on Linux, we hit a huge bottleneck waiting for the FULL commit to happen for each added package during reporead operations. It makes much more sense to back this off to FULL level instead, which trades some possible loss of durability for speedier operation. Additionally, no one would possibly be running their production version of this site on sqlite3, right? Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-06reporead: don't append slash to empty (root) directoryDan McGee1-1/+2
Add the slash only if we have a directory name, and not otherwise. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-05reporead: handle files in root directory properlyDan McGee1-1/+4
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-05reporead: properly handle cases where last_update == files_last_updateDan McGee1-2/+2
We should assume the filelists are up to date in this case, not out of date. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-07-02Log package updates during reporead invocationDan McGee1-1/+6
This adds a Manager and log_update method to help log all updates made to the packages table during reporead runs. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-19reporead: fix copy/paste issueDan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-19Switch to usage of new Depend objectDan McGee1-8/+10
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-05-02rematch_developers: do mass updates instead of single savesDan McGee1-31/+30
When updating a lot of objects, it makes much more sense to perform targeted update queries rather than one-row-at-a-time saves. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-04-20UserProfile model and fields shuffleDan McGee1-2/+1
Move this model into the devel/ application, and move the PGPKeyField which is used only by these models into the application as well. This involves updating some old migrations along the way to ensure we don't reference a field class that no longer exists. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-26Rename 'packagedepend_set' attribute to 'depends'Dan McGee1-1/+1
We do this for every other related package attribute, so do it here too. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24reporead: use bulk_create() for more propertiesDan McGee1-13/+17
Depends, conflicts, provides, etc. can all be done via bulk_create. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24Merge branch 'django14'Dan McGee1-23/+13
Conflicts: templates/releng/result_section.html
2012-03-24Make all datetime objects fully timezone awareDan McGee1-4/+8
This is most of the transition to Django 1.4 `USE_TZ = True`. We need to ensure we don't mix aware and non-aware datetime objects when dealing with datetimes in the code. Add a utc_now() helper method that we can use most places, and ensure there is always a timezone attached when necessary. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24reporead: use Django 1.4 bulk_create() for package filesDan McGee1-4/+3
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24reporead: use Django 1.4 select_for_update()Dan McGee1-15/+2
As per TODO comments in the existing code. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-24reporead: blow up when package found with wrong architectureDan McGee1-2/+3
Signed-off-by: Dan McGee <dan@archlinux.org>
2012-03-16reporead: rename Pkg to RepoPackageDan McGee1-3/+4
The bytes saved on the shorter name aren't worth it. Also ensure 'desc' is always initialized to None in case packages do not provide one. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-02-22import_signatures: code cleanup and ensure we update validity statusDan McGee1-10/+29
This adds a namedtuple so we aren't using magic numbers when processing our 'edges' (signatures). We also ensure we update any existing signature objects with their validity if they were later revoked. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-02-12reporead: only reset flag date if upstream version changesDan McGee1-1/+7
This preserves the flag date if only a simple pkgrel bump occurred, which makes sense more often than not for rebuilds. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-02-07Make rematch command do a bit moreDan McGee1-7/+40
Now that we have a few objects that can potentially link back to developers, allow flag requests to also be rematched. Signed-off-by: Dan McGee <dan@archlinux.org>
2012-01-19reporead: simplify and fix transaction management in update_common()Dan McGee1-15/+16
We can use the easier transaction.commit_on_success() decorator if we be sure to explicitly mark the transaction dirty. This fixes the issue where a raised exception in this code called neither commit nor rollback. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-12reporead: more efficient deletion of filesDan McGee1-1/+9
Rather than delegating to Django and batch deletion by ID, force issuing of a single delete query to clear out all existing file objects when necessary. This should speed up the deletion and update of packages with a lot of files by a non-trivial amount. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-12PyLint suggested cleanupsDan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-06Add a stub admin command to help fix permissions/content typesDan McGee1-0/+25
I needed this today to get the application working from scratch on another host. Probably not all there yet, but we'll see how far it gets us. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-06Add import signatures management commandDan McGee1-0/+104
This allow importing signatures from a provided gpg keyring, such as that produced by the generate_keyring management command that already exists. These will eventually be used for producing stats involving developer signing keys and their certification by master keys. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-03reporead: don't update timestamp on --forceDan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-03reporead: fix --force flagDan McGee1-5/+4
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-12-01reporead: fix not defined variableDan McGee1-0/+2
Way to fail at refactoring, Dan. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-30Allow generation of an ownertrust fileDan McGee1-3/+26
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-30Integrate master key into rest of siteDan McGee1-0/+4
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-30reporead: split out filesonly update methodDan McGee1-75/+95
This removes a bunch of the conditional logic at a slight cost of some code duplication. However, the methods and madness is now much easier to follow. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-30reporead: fix filesonly needs update checksDan McGee1-3/+5
This was broken after the select for update changes. We really should split the whole filesonly update into another method instead of the current shotgun approach with conditionals everywhere. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-17reporead: don't trim pkgdesc lengthDan McGee1-3/+3
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-17Ensure reporead is protected against simultaneous runsDan McGee1-100/+106
This adds a bunch of transaction magic and SELECT FOR UPDATE stuff to reporead to cope with the now-concurrent runs of reporead we get when invoked from our inotify-based updater. The collision occurs with 'any' architecture packages as both repo databases contain the new version, and the updates occur at exactly the same time. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-17reporead_inotify: nice the spawned subprocessesDan McGee1-4/+9
This prevents the reporead job from taking over time from more important processes; this is not a rush task. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16reporead_inotify: close connection once we are done with itDan McGee1-0/+6
This prevents an otherwise idle connection from sitting around and being totally useless. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16reporead_inotify: spin up read_repo() in separate threadDan McGee1-2/+9
This prevents memory usage from ballooning to absolutely huge values, such as when multiple threads kick off at the same time. The bulk of our memory allocation obviously comes in these threads and not the main threads, so being able to isolate them in processes helps a lot. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16reporead: a few small tweaksDan McGee1-3/+4
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16reporead: clean up some debug loggingDan McGee1-3/+5
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16reporead_inotify: cancel threads that haven't started yet on shutdownDan McGee1-0/+5
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16Add new reporead_inotify management commandDan McGee1-0/+188
This is the new on-the-fly updates hotness. Rather than continue to schedule reporead to run once an hour in cron or however else you ran it, this command can be run once and left running, and will automagically pick up on any database file changes and run an import. It operates on the files databases only; this will keep both the packages and files always in sync and remove the delay in updating, especially helpful for new testing packages. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-16Improve primary arch validationDan McGee1-7/+14
Ensure we can accept either a Arch object or an architecture name when passed to read_repo() by moving the validation there and being a bit more careful about typechecking and object lookup. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-11-01Really ensure we don't catch any NULL or blank valuesDan McGee1-4/+4
Fuck you too, Django. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-26Ensure PGP signature values are not trimmedDan McGee1-1/+5
This makes them totally unusable for any real purpose down the road. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-10-22Add a generate_keyring commandDan McGee2-1/+60
This grabs all the PGP keys from the developer profiles and adds them to the keyrings. Obviously we may want to do more in the future such as filter by groups, active status, etc. but this is just a first iteration. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-17Add PGP signature package fieldDan McGee1-0/+1
And add eventual display code for it to the details template, but don't show it yet as no packages will have it. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-08-16Add two new DB fields to reporeadDan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24Add a rematch_packager management commandDan McGee1-0/+64
This allows quick resolution of all unmatched packages, especially after tweaking the way find_user works. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-06-24Turn find_user into UserFinder classDan McGee1-2/+4
This moves the cache inside an instance. Also add a few more tests. Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-24-g4f1b (git 2.32.0) at 2024-11-06 14:29:24 +0100