summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2020-05-09Convert downloading databases to the new multiplexed APIAnatol Pomozov1-18/+9
Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Convert download packages logic to multiplexed APIAnatol Pomozov5-62/+36
Create a list of dload_payloads and pass it to the new _alpm_multi_* interface. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Implement multibar UIAnatol Pomozov8-159/+320
Multiplexed download requires ability to draw UI for multiple active progress bars. To implement it we use ANSI codes to move cursor up/down and then redraw the required progress bar. `pacman_multibar_ui.active_downloads` field represents the list of active downloads that correspond to progress bars. `struct pacman_progress_bar` is a data structure for a progress bar. In some cases (e.g. database downloads) we want to keep progress bars in order. In some other cases (package downloads) we want to move completed items to the top of the screen. Function `multibar_move_completed_up` allows to configure such behavior. Per discussion in the maillist we do not want to show download progress for signature files. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Extend download callback interface with start/complete eventsAnatol Pomozov5-10/+59
With the previous download interface the callback uses the first progress event as 'download has started' signal. Unfortunately it does not work with up-to-date files that never receive 'download progress' events. Up-to-date database messages are currently handled in sync_syncdbs() after the sequential download is completed and a result from ALPM is received. But this is not going to work with multiplexed download interface that returns the result only after all files are completed. Another problem with 'first progress event is the beginning of the download' is that such events time are unpredictable. Thus the UI progress bar order might differ from what has been passed by client to alpm_dbs_update() function. We actually want to keep the dbs progress bars in a strict order. To help to solve the given problems extend the download callback to allow 2 more events - download started and completed. 'Download started' events appear in the same order as in the list given by a client. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Introduce event types for start/end database list downloadAnatol Pomozov4-12/+34
Multiplexed database/files downloads will use multiple progress bars. The UI logic is quite complicated and printing error messages while handling multiple progress bars is going to be challenging. Instead we are going to save all ALPM error messages to a list and flush it at the end of the download process. Use on_progress variable that blocks error messages printing. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Implement multiplexed download using mCURLAnatol Pomozov2-10/+388
curl_multi_download_internal() is the main loop that creates up to 'ParallelDownloads' easy curl handles, adds them to mcurl and then performs curl execution. This is when the paralled downloads happens. Once any of the downloads complete the function checks its result. In case if the download fails it initiates retry with the next server from payload->servers list. At the download completion all the payload resources are cleaned up. curl_multi_check_finished_download() is essentially refactored version of curl_download_internal() adopted for multi_curl. Once mcurl porting is complete curl_download_internal() will be removed. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Implement _alpm_multi_downloadAnatol Pomozov1-4/+46
It is an equivalent of _alpm_download but accepts a list of payloads. curl_multi_download_internal() is a stub at this moment and will be implemented in the later commits of this patch series. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Inline dload_payload->curlerr field into a local variableAnatol Pomozov2-5/+5
dload_payload->curlerr is a field that is used inside curl_download_internal() function only. It can be converted to a local variable. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Add multi_curl handle to ALPM global contextAnatol Pomozov5-20/+17
To be able to run multiple download in parallel efficiently we need to use curl_multi interface [1]. It introduces a set of APIs over new type of handler 'CURLM'. Create CURLM object at the application start and set it to global ALPM context. The 'single-download' CURL handle moves to payload struct. A new CURL handle is created for each payload with intention to be processed by CURLM. Note that curl_download_internal() is not ported to CURLM interface due to the fact that the function will go away soon. [1] https://curl.haxx.se/libcurl/c/libcurl-multi.html Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Introduce alpm_dbs_update() function for parallel db updatesAnatol Pomozov4-0/+178
This is an equivalent of alpm_db_update but for multiplexed (parallel) download. The difference is that this function accepts list of databases to update. And then ALPM internals download it in parallel if possible. Add a stub for _alpm_multi_download the function that will do parallel payloads downloads in the future. Introduce dload_payload->filepath field that contains url path to the file we download. It is like fileurl field but does not contain protocol/server part. The rationale for having this field is that with the curl multidownload the server retry logic is going to move to a curl callback. And the callback needs to be able to reconstruct the 'next' fileurl. One will be able to do it by getting the next server url from 'servers' list and then concat with filepath. Once the 'parallel download' refactoring is over 'fileurl' field will go away. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09Add config option to specify amount of parallel download streamsAnatol Pomozov7-0/+89
It includes pacman.conf new 'ParallelDownloads' option that specifies how many concurrent downloads cURL starts in parallel. Add alpm_option_set_parallel_downloads() ALPM function that allows to set this config option programmatically. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-09doc: remove vim modelines from BUILDINFO(5)Eli Schwartz1-3/+0
We (thought we) removed all modelines from the project in commit 860e4c4943ad062bd0eff99f28e7d64804b3c08e, but apparently this one sneaked in by virtue of this manpage being added to the project after the "remove all the modelines" patch was submitted, but before it was applied. I must have failed to update the patch to remove it from this file also. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-06editorconfig: set meson indentation styleEli Schwartz1-0/+4
meson.build gets two-space indents, but our global tabbed default was overriding this. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-06meson: use better check for debug buildsEli Schwartz1-1/+1
meson 0.48 added the 'debug' and 'optimization' builtin options, which bidirectionally map to the buildtype, but in some cases where debug is enabled, the builtype may be custom. Checking the 'debug' option lets us detect every case currently detected, plus a few more, and does so in a shorter and more concise manner. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-06Convert gitlab valgrind runner to mesonAllan McRae1-5/+4
Turns out environmental variables do get passed through fakechroot! Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-05meson: handle XFAIL tests outside of TAPDave Reisner2-2/+20
This change causes expected fail tests to actually fail by eliding the '# TODO' from the test plan. In turn, we can now properly use 'should_fail' in the meson test() rule and see these expected fail tests in the output: Before: ... 320/332 upgrade077.py OK 0.12679290771484375 s 321/332 upgrade078.py OK 0.12620115280151367 s 322/332 upgrade080.py OK 0.1252129077911377 s ... Ok: 332 Expected Fail: 0 Fail: 0 Unexpected Pass: 0 Skipped: 0 Timeout: 0 After: ... 320/332 upgrade077.py OK 0.12679290771484375 s 321/332 upgrade078.py EXPECTEDFAIL0.12620115280151367 s 322/332 upgrade080.py OK 0.1252129077911377 s ... Ok: 326 Expected Fail: 6 Fail: 0 Unexpected Pass: 0 Skipped: 0 Timeout: 0 Signed-off-by: Allan McRae <allan@archlinux.org>
2020-05-05Add Gitlab Runner configurationAllan McRae1-0/+128
Arch Linux is setting up a Gitlab instance. This adds CI for the pacman project, testing a range of configurations on Arch and basic builds on Fedora and Debian. Note that asciidoc is specifically not installed on the Debian run because it is all sorts of broken... Also, the defaults have been set to meson, with two autotools tests that will soon be removed. Original-file from: Andrew Gregory <andrew.gregory.8@gmail.com> Altered-to-run-on-Arch-Gitlab by: Sven-Hendrik Haase <svenstaro@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-29Add NoProgressbar to pacman.conf optionsIvy Foster4-0/+10
This is useful for dumb terminals that do not support escape sequences. Signed-off-by: Ivy Foster <escondida@iff.ink> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-29Constify some input pointersRikard Falkeborn4-4/+4
Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-15pacman.8: Fix typoAllan McRae1-1/+1
Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-15Dull version colour numbers in summaryCarson Black3-12/+39
Version colour numbers are dulled in the non-verbose transaction summary when colours are enabled. To prevent a regression, this patch also adds handling of strings with ANSI codes to string_length as to not break the transaction summary's output functions when colour codes are in the package name strings. Signed-off-by: Carson Black <uhhadd@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Use GOTO_ERR throughoutAllan McRae6-41/+22
The GOTO_ERR define was added in commit 80ae8014 for use in future commits. There are plenty of places in the code base it can be used, so convert them. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Use STRDUP for error checking in more placesAllan McRae3-5/+10
Use STRDUP() over strdup() to catch memory allocation errors. There are still some instances of strdup left, but these are in functions that currently have no error path and would require a larger rework. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Add REALLOC macro to simplify realloc error handlingRikard Falkeborn5-19/+7
realloc can fail just like the other memory allocation functions. Add a macro to simplify handling of realloc failures, similar to the already existing MALLOC, CALLOC, etc. Replace the existing realloc uses with the new macro, allowing us to move tedious error handling to the macro. Also, in be_package and be_sync, this fixes hypothetical memory leaks (and thereafter null pointer dereferences) in case realloc fails to shrink the allocated memory. Signed-off-by: Rikard Falkeborn <rikard.falkeborn@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-04-13Use noextract with pacman-conf NoExtractEarnestly1-1/+1
Current code accidently uses noupgrade for the NoExtract directive. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-13Hide cursor while pacman is runningAnatol Pomozov4-5/+38
Use ASCII control codes to hide cursor at the pacman start and then show the cursor when pacman finishes. It helps to avoid annoying blinking when progress bars are re-drawn. Cursor is reenabled if pacman expects user's input. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-13Add file and line number to RET_ERR{,_VOID}Allan McRae1-2/+2
Following the example of the recently added GOTO_ERR, adding the file and line number in addition to the function name in our debug messages is potentially useful. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-09Add GOTO_ERR() macro to set error and then goto a labelAnatol Pomozov1-0/+5
This is a macro similar to RET_ERR but useful in the case when we need to record an error and then jump to some cleanup section. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-09Remove unneeded goto cleanupAllan McRae1-11/+5
Since commit 2ee7a8d8, there is no cleanup needed in this function. Just return instead of jumping to the cleanup label. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-06Move flushing 'output' list into separate functionAnatol Pomozov1-7/+11
'output' is a list of messages that pacman received but delayed printing to avoid messing with UI. Such functionality is useful for the upcoming multi-line progress bar UI. Let's move it to a separate function. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-03-05Do not download files if find_dl_candidates() failsAnatol Pomozov1-1/+1
One reason why the function returns an error is some repo does not have any servers. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-02-24Simplify construction of payloads in download_filesAnatol Pomozov1-21/+13
Currently, download_files() creates payloads for all packages then iterates over them, calling download_single_file. This can be simplified by looping over packages and constructing the payload as needed. Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-02-12makepkg: drop duplicate reporting of missing dependenciesDave Reisner1-6/+0
When pacman fails to satisfy deps, we might see output like the following: ==> Making package: spiderfoot 3.0-1 (Thu 06 Feb 2020 12:45:10 PM CET) ==> Checking runtime dependencies... ==> Installing missing dependencies... error: target not found: python-pygexf ==> ERROR: 'pacman' failed to install missing dependencies. ==> Missing dependencies: -> python-dnspython -> python-exifread -> python-cherrypy -> python-beautifulsoup4 -> python-netaddr -> python-pysocks -> python-ipwhois -> python-ipaddress -> python-phonenumbers -> python-pypdf2 -> python-stem -> python-whois -> python-future -> python-pyopenssl -> python-docx -> python-pptx -> python-networkx -> python-cryptography -> python-secure -> python-pygexf -> python-adblockparser ==> Checking buildtime dependencies... ==> ERROR: Could not resolve all dependencies. This is misleading -- the only truly missing package is python-pygexf, but we fail to remove sync-able deps from our deplist and report everything as if it were missing. Simply drop this extra reporting because pacman already tells us exactly what couldn't be resolved. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-02-10build-aux/update-copyright 2019 2020Allan McRae179-185/+185
Signed-off-by: Allan McRae <allan@archlinux.org>
2020-02-10Add pacman-hooks.5 man page symlinkAllan McRae2-1/+16
Improve discoverability of the alpm-hooks man page by adding a pacman-hooks symlink. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-30Eliminate extra loop over dbs_syncAnatol Pomozov1-9/+5
Current flow looks like loop dbs_sync { loop pkgs { if pkg.db == db then process(pkg, db) } } Package sync transaction always has a counterpart in the dbs_sync list (I cannot come up with a use-case when it is not true). So the loop can be simplified to: loop pkgs { process(pkg, pkg.db) } Tested: 'ninja test' & manually by using pacman with this patch for a week Signed-off-by: Anatol Pomozov <anatol.pomozov@gmail.com> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-28Docs docs docsmorganamilo17-298/+241
libalpm: move docs from .c files into alpm.h And fix/expand some along the way. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-28libalpm: fix alpm_option_set_assumeinstalledmorganamilo1-0/+1
It looks like this function has never actually worked. The current list is never set to NULL after being freed. So the new deps were just appended to the already freed list, leading to a segfault. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-28makepkg: add CRC checksums and set these to be the defaultAllan McRae5-13/+13
Checksums arrays should be filled with values provided by upstream. We currently have md5 set as an unsecure default, and are constantly asked to change it to sha2. However, just changing the default to a stronger checksum gives the user the impression that "makepkg -g" checksums are perfect. Instead, change the default checksum to a CRC, to make it clear that any checksum generated purely by "makepkg -g" is not ideal. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-28Note that checksums from "makepkg -g" are not idealAllan McRae1-1/+3
Generating checksums with "makepkg -g" only determines that the user of a PKGBUILD has the same file as the packager (assuming no collision). This means an upstream source could be maliciously changed and passed on as valid by a PKGBUILD. To avoid this, it is essential that any checksums used in a PKGBUILD are as provided by upstream. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27Fix "pacman -U <url>" operationsAllan McRae1-1/+1
Commit e6a6d307 detected complete part files by comparing a payload's max_size to initial_size. However, these values are also equal when we use pacman -U on a URL as max_size is set to 0 in that case. Add a further condition to avoid that. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27makepkg: make per-package files containing '$pkgname' consistently workEli Schwartz3-9/+21
Extracting function variables containing arbitrarily scoped variables of arbitrary nature is a disaster, but let's at least cover the common case of using the actual '$pkgname' in an install/changelog file. It's the odd case of actually being basically justified use of disambiguating between the same variable used in multiple different split packages... and also, --printsrcinfo already uses and overwrites the variable 'pkgname' in pkgbuild_extract_to_srcinfo, so this "works" in .SRCINFO but doesn't work in .src.tar.gz It doesn't work in lint_pkgbuild either, but in that case the problem is being too permissive, not too restrictive -- we might end up checking the same file twice, and printing that it is missing twice. Fixes FS#64932 Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27Increase maximum database sizeAllan McRae1-2/+2
We previously has the maximum database size as 25MB. This was set in the days before repos had as many packages as they do now, and before we started distributing files databases. Increase this limit to 128MB. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27Remove unneeded ltmain patchAllan McRae2-39/+0
Many moons ago, libtool was bad - I mean worse than today! It gobbled all --as-needed and we ended up with an overlinked libalpm. This was annoying, particularly when dealing with soname bumps in libraries pacman/libalpm had no business linking to. Luckily we had a fix, stolen from GNOME I believe. And with that fix, we lived in harmony with libtool for many years. Until one day, unbeknownst to us, libtool was "fixed". We kept applying our patch, because it still applied, but it did worse than nothing. It gobbled up our other LDFLAGS, and our libalpm started missing out on RELRO and BIND_NOW. This made the Arch Security Team unhappy. We will make them happy again by stopping the patch. Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27Increase meson test timeoutAllan McRae1-0/+1
Running the testsuite using "PACTEST_VALGRIND=1 ninja test -C build", I ran into the following failure: 161/332 smoke001.py TIMEOUT 30.02 s I figure an i7 @ 3.10GHz should be enough to run our testsuite... so boost the meson test timeout to 120 seconds (which should be enough time for anyone...). Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-27tests: Fix typo in test descriptionAllan McRae1-1/+1
2020-01-07doc: reformat intro to VCS sources to distinguish the grammarEli Schwartz1-5/+7
It's difficult to find it embedded inside a prose paragraph. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-07doc: clarify the format of a PKGBUILD source fragmentEli Schwartz1-3/+3
Currently, it could be misread to say that a fragment is literally 'commit', rather than 'commit=somehash'. Anecdotally this does not seem to be obvious to everyone, and rewording it certainly doesn't hurt. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-07autotools: emit error message when autoconf-archive is missingEli Schwartz1-0/+1
Forbid the AX_COMPARE_VERSION macro from being found in the output configure script. If autoconf-archive is not installed when autoreconf is run, the following error message is emitted: configure.ac:231: error: possibly undefined macro: AX_COMPARE_VERSION If this token and others are legitimate, please use m4_pattern_allow. See the Autoconf documentation. autoreconf: /usr/bin/autoconf failed with exit status: 1 Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org>
2020-01-07Use c99 struct initialization to avoid memset callsDave Reisner7-40/+20
This is guaranteed less error prone than calling memset and hoping the human gets the argument order correct.
generated by cgit v1.2.3-24-g4f1b (git 2.32.0) at 2024-06-03 08:37:07 +0200