summaryrefslogtreecommitdiffstats
path: root/lib/libalpm/dload.c
AgeCommit message (Collapse)AuthorFilesLines
2011-04-23libalpm/dload: major refactor of signature downloadingDan McGee1-65/+36
There's a lot of related moving parts here: * Iteration through mirrors is moved back to the calling functions. This allows removal of _alpm_download_single_file and _alpm_download_files. * The download function gets a few more arguments to influence behavior. This allows several different scenarios to customize behavior: - database - database signature (req'd and optional) - package - package via direct URL - package signature via direct URL (req'd and optional) * For databases, we need signatures from the same mirror, so structure the code accordingly. Some-inspiration-from: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-23libalpm/dload: add allow_resume and reorder error checksDan McGee1-19/+21
The allow_resume is the start of the fix to the "don't ever resume database downloads" problem, as well as being useful for '.sig' downloads as well. For now, we say "always allow resume", but this will eventually get pushed down as necessary. Error checks are reworked in order to correctly error out when a file is not found on the remote end and reports 0 bytes downloaded. In addition, the two error messages printed are now different as one reports a more specific error message provided via the cURL error buffer. Some example output from an -Sy run with [testing], [community], [community2], [eee], and [nonexistant] defined as repos. [community2] and [nonexistant] are both invalid, one using FTP and one using HTTP. :: Synchronizing package databases... testing is up to date community is up to date error: failed retrieving file 'community2.db' from ftp.archlinux.org : Given file does not exist error: failed to update community2 (FTP: couldn't retrieve (RETR failed) the specified file) eee is up to date error: failed retrieving file 'nonexistant.db' from code.toofishes.net : The requested URL returned error: 404 error: failed to update nonexistant (HTTP response code said error) Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-22libalpm/dload: const and static correctnessDan McGee1-5/+6
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-21syntax: if/while statements should have no trailing spaceDan McGee1-1/+1
This is the standard, and we have had a few of these introduced lately that should not be here. Done with: find -name '*.c' | xargs sed -i -e 's#if (#if(#g' find -name '*.c' | xargs sed -i -e 's#while (#while(#g' Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-21style cleanup: cast as (type *) not (type*)Dave Reisner1-1/+1
Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-16Merge branch 'maint'Dan McGee1-5/+1
Conflicts: lib/libalpm/alpm.h lib/libalpm/trans.c Signed-off-by: Dan McGee <dan@archlinux.org>
2011-04-09Move documentation for public package function to alpm.hRémy Oudompheng1-5/+1
2011-04-02dload: dont forget to initialize open_modeXavier Chantry1-1/+2
That's a funny one, building with optimization levels (with both gcc and clang) caused open_mode to always be set to "ab", which worked. This was spotted both with clang-analyzer, and by Jakob who reported a segfault as he was using an un-optimized build. Signed-off-by: Xavier Chantry <chantry.xavier@gmail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28lib/dload: abstract out helper function to set utimesDave Reisner1-22/+22
This greatly simplifies the cleanup fallthrough in our download function and we'll be able to reuse this for signatures. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28lib/dload: remove proxy debug outputDave Reisner1-6/+0
Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28lib/dload: code simplificationDave Reisner1-11/+8
Based on the fact that localf always points to the same file, there's no need to code in multiple fopen calls with varying results. Instead, track the desired file open mode and make a single call to fopen. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28lib/dload: merge get_{destfile,tempfile} into get_fullpathDave Reisner1-20/+13
Create a more general function that allows appending a suffix to a filepath. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28pacman/pacman.c: fix setting of useragent stringDave Reisner1-1/+6
libcurl doesn't natively honor the HTTP_USER_AGENT environment variable. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-28lib/dload: pass a struct with filename and size to curl_progressDave Reisner1-13/+21
This lets us determine the real size of the file on disk so that we can properly bump the progress bar when we're resuming a download. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23lib/dload.c: Check for dlcb == NULL earlierDave Reisner1-10/+12
Our curl callback does a whole lot of work for nothing if the front end never defined a callback to receive the data we'd calculate for it. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23lib/dload.c: don't use deprecated curl symbolsDave Reisner1-1/+1
CURLINFO_HTTP_CODE is deprecated in favor of CURLINFO_RESPONSE_CODE. Both yield the same values. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23lib/dload.c: don't request compressed transportDave Reisner1-1/+0
The files we transfer are generally compressed already, so this just adds unnecessary overhead. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23lib/dload.c: Fix progress callback issues on downloadDave Reisner1-5/+15
Use a static variable to effectively track the initialization state of the progress callback via the last byte amount reported as downloaded by libcurl. Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-23lib/dload.c: fix compiler warnings generated by -Wfloat-equalDave Reisner1-3/+4
* introduces new macro in util.h (DOUBLE_EQ) for properly comparing floating point values Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-21Don't initialize progress to zero before calling curl_easy_perform().Lukas Fleischer1-4/+3
Drawing progress bars before calling curl_easy_perform() is needless as the curl progress callback is called with zero progress before actually downloading the file anyways. Fixes display of "0%" progress bars when sync'ing package databases that are already up to date. Signed-off-by: Lukas Fleischer <archlinux@cryptocrack.de> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-21Ensure dlcb is defined before calling itDan McGee1-1/+3
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-21Style change: return(x) --> return xDan McGee1-16/+16
This was discussed and more or less agreed upon on the mailing list. A huge checkin, but if we just do it and let people adjust the pain will end soon enough. Rebasing should be relatively straighforward for anyone that sees conflicts; just be sure you use the new return style if possible. The following semantic patch was used to do the change, along with some hand-massaging in order to preserve parenthesis where appropriate: The semantic match that finds this problem is as follows, although some hand-massaging was done in order to keep parenthesis where appropriate: (http://coccinelle.lip6.fr/) // <smpl> @@ expression a; @@ - return(a); + return a; // </smpl> A macros_file was also provided with the following content: Additional steps taken, mainly for ASSERT() macros: $ sed -i -e 's#return(NULL)#return NULL#' lib/libalpm/*.c $ sed -i -e 's#return(-1)#return -1#' lib/libalpm/*.c Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-21lib/dload.c: fix opening braces to conform with coding styleDave Reisner1-6/+12
Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-20lib/dload.c: remove lingering libfetch specific headersDave Reisner1-5/+0
Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-17Fix triple progress bars on downloadDave Reisner1-1/+5
Signed-off-by: Dave Reisner <d@falconindy.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2011-03-09Remove all traces of libfetchDave Reisner1-282/+2
Signed-off-by: Dave Reisner <d@falconindy.com>
2011-03-09dload: temp patch to allow curl/fetch coexistanceDave Reisner1-1/+15
this is just some debuggery to allow pacman to operate with both fetch and curl at the same time. use the PACMANDL variable to control which library is used. Signed-off-by: Dave Reisner <d@falconindy.com>
2011-03-09dload.c: add curl_download_internalDave Reisner1-0/+208
This is a feature complete re-implementation of the fetch based internal downloader, with a few improvements: * support for SSL * gzip and deflate compression on HTTP connections * reuses a single connection over the entire session for lower resource usage. Signed-off-by: Dave Reisner <d@falconindy.com>
2011-03-09share code between libfetch and libcurlDave Reisner1-10/+12
no actual code changes here. change preprocessor logic to include get_tempfile, get_destfile, signal handler enum, and the interrupt handler logic when either HAVE_LIBCURL or HAVE_LIBFETCH are defined. Signed-off-by: Dave Reisner <d@falconindy.com>
2011-03-09prefix fetch based functions with fetch_Dave Reisner1-6/+6
Do this in preparation for implementing similar curl based functionality. We want the ability to test these side by side. Signed-off-by: Dave Reisner <d@falconindy.com>
2011-03-09add curl to alpm initialization and teardown routinesDave Reisner1-0/+4
Signed-off-by: Dave Reisner <d@falconindy.com>
2011-02-04Add comment about download file resolutionDan McGee1-0/+8
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-04Add more error checking and loggingDan McGee1-1/+5
Signed-off-by: Dan McGee <dan@archlinux.org>
2011-02-04Ensure found files are actually filesDan McGee1-2/+2
We located files in a few places but didn't check if they were files or directories. Ensure they are actually files using stat() and S_ISREG(); this showed itself when trying to download to the directory name itself in FS#22645. Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-08Fix some more simple conversion "errors"Dan McGee1-1/+1
None of these warn at the normal "-Wall -Werror" level, but casts do occur that we are fine with. Make them explicit to silence some warnings when using "-Wconversion". Signed-off-by: Dan McGee <dan@archlinux.org>
2011-01-08Update copyright years for 2011Allan McRae1-1/+1
Signed-off-by: Allan McRae <allan@archlinux.org> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-30Use limits.h for PATH_MAXAllan McRae1-4/+2
We use PATH_MAX everywhere by including limits.h so there is no point in doing a check for it in a different header when dealing with FreeBSD's libfetch. Also, remove autoconf check for strings.h header as it is not used anywhere. Signed-off-by: Allan McRae <allan@archlinux.org> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-30Detect undefined PATH_MAXAllan McRae1-1/+0
POSIX does not require PATH_MAX be defined when there is not actual limit to its value. This affects HURD based systems. Work around this by defining PATH_MAX to 4096 (as on Linux) when this is not defined. Also, clean up inclusions of limits.h and remove autoconf check for this header as we do not use macro shields for its inclusion anyway. Signed-off-by: Allan McRae <allan@archlinux.org> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-12-13Remove AC_TYPE_SIGNAL usageDan McGee1-1/+1
This macro is deemed unnecessary by even the autoconf guys, so we really don't need to use it. Signed-off-by: Dan McGee <dan@archlinux.org> Signed-off-by: Allan McRae <allan@archlinux.org> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-06Fix libfetch configure checkingDan McGee1-3/+3
I don't know what I tested in commit 3e7b90ff6950, but it definitely wasn't working as advertised. Fix the checks in the source code itself to match the right define (HAVE_LIBFETCH), as well as make sure the configure check defaults to looking for the library but not bailing if it could not be found. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-09-02Clean up libfetch checking in configureDan McGee1-3/+3
Model it after the new OpenSSL check, and have it be a bit more useful. If you do not explicitly pass a command line option, it will be linked if available but will not error out if it is missing. Also bump the version to that where connection caching was introduced as we use these new features in the codebase. Signed-off-by: Dan McGee <dan@archlinux.org>
2010-06-30Do not compare signed and unsigned typesAllan McRae1-1/+1
The casting of nread is safe as it is tested to be >0 when it is initally assigned. It is also being implicitly cast in the fwrite call in the line above. Signed-off-by: Allan McRae <allan@archlinux.org> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-03-26More consistent printing of off_t and time_tXavier Chantry1-3/+3
time_t : %ld off_t : %jd and cast to intmax_t Signed-off-by: Xavier Chantry <shiningxc@gmail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2010-03-15Bump copyright dates to 2010Dan McGee1-1/+1
Signed-off-by: Dan McGee <dan@archlinux.org>
2009-12-14improve download_internal error messagesXavier Chantry1-2/+8
download_internal is supposed to always set pm_errno but did not in many cases. The most important (and tested) change is the one concerning fetchStat. This is typically where the code will fail when the network is down for example. Before commit d2dbb04a9af7a18da, this fetchStat call did not exist and the same kind of errors would be encountered in the fetchXGet call that follows. I just copied the error printing to restore the old behavior. Signed-off-by: Xavier Chantry <shiningxc@gmail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2009-11-16download: major refactor to address lingering issuesDan McGee1-56/+121
Sorry for this being such a huge patch, but I believe it is necessary for quite a few reasons which I will attempt to explain herein. I've been mulling this over for a while, but wasn't super happy with making the download interface more complex. Instead, if we carefully order things in the internal download code, we can actually make the interface simpler. 1. FS#15657 - This involves `name.db.tar.gz.part` files being left around the filesystem, and then causing all sorts of issues when someone attempts to rerun the operation they canceled. We need to ensure that if we resume a download, we are resuming it on exactly the same file; if we cannot be almost postive of that then we need to start over. 2. http://www.mail-archive.com/pacman-dev@archlinux.org/msg03536.html - Here we have a lighttpd bug to ruin the day. If we send both a Range: header and If-Modified-Since: header across the wire in a GET request, lighttpd doesn't do what we want in several cases. If the file hadn't been modified, it returns a '304 Not Modified' instead of a '206 Partial Content'. We need to do a stat (e.g. HEAD in HTTP terms) operation here, and the proceed accordingly based off the values we get back from it. 3. The mtime stuff was rather ugly, and relied on the called function to write back to a passed in reference, which isn't the greatest. Instead, use the power of the filesystem to contain this info. Every file downloaded internally is now carefully timestamped with the remote file time. This should allow the resume logic to work. In order to guarantee this, we need to implement a signal handler that catches interrupts, notifies the running code, and causes it to set the mtimes on the file. It then rethrows the signal so the pacman signal handler (or any frontend) works as expected. 4. We did a lot of funky stuff in trying to track the DB last modified time. It is a lot easier to just keep the downloaded DB file around and track the time on that rather than in a funky dot file. It also kills a lot of code. 5. For GPG verification of the databases down the road, we are going to need the DB file around for at least a short bit of time anyway, so this gets us closer to that. Signed-off-by: Dan McGee <dan@archlinux.org> [Xav: fixed printf with off_t] Signed-off-by: Xavier Chantry <shiningxc@gmail.com>
2009-10-21dload.c : clear sigaction flag to make valgrind happyXavier Chantry1-0/+1
This fixes the following valgrind warning : ==26831== Syscall param rt_sigaction(act->sa_flags) points to uninitialised byte(s) ==26831== at 0x4282547: __libc_sigaction (in /lib/libc-2.10.1.so) ==26831== by 0x403C693: download_internal (dload.c:152) ==26831== by 0x403D0E4: _alpm_download_single_file (dload.c:311) ==26831== by 0x4033B72: alpm_db_update (be_files.c:319) ==26831== by 0x805205E: pacman_sync (sync.c:257) ==26831== by 0x804EE54: main (pacman.c:1120) ==26831== Address 0xbec6cc04 is on thread 1's stack ==26831== ==26831== Syscall param rt_sigaction(act->sa_restorer) points to uninitialised byte(s) ==26831== at 0x4282547: __libc_sigaction (in /lib/libc-2.10.1.so) ==26831== by 0x403C693: download_internal (dload.c:152) ==26831== by 0x403D0E4: _alpm_download_single_file (dload.c:311) ==26831== by 0x4033B72: alpm_db_update (be_files.c:319) ==26831== by 0x805205E: pacman_sync (sync.c:257) ==26831== by 0x804EE54: main (pacman.c:1120) ==26831== Address 0xbec6cc08 is on thread 1's stack ==26831== Signed-off-by: Xavier Chantry <shiningxc@gmail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
2009-10-11typing: a few more fixes for special int typesDan McGee1-4/+4
Signed-off-by: Dan McGee <dan@archlinux.org>
2009-09-30Merge branch 'maint'Dan McGee1-7/+22
Conflicts: lib/libalpm/dload.c
2009-09-30dload : fix infinite download (big type mistake)Xavier Chantry1-7/+22
fetchIO_read returns -1 in case of error, and the return type is ssize_t, not size_t ! So we converted -1 to an unsigned, which led to huge file write. The rest is just changing the error return a bit. Signed-off-by: Xavier Chantry <shiningxc@gmail.com> Signed-off-by: Dan McGee <dan@archlinux.org>
generated by cgit v1.2.3-24-g4f1b (git 2.32.0) at 2024-06-02 23:08:08 +0200