Parallel download with R's native libcurl

[noamross]

R-devel (4.5) uses internal libcurl's simultaneous downloads capability to install many packages very quickly with install.packages, similar to pak. Will renv be able to take advantage of this to parallelize package downloads?

[kevinushey]

Will this functionality be extended to download.file()? It would take a somewhat substantial amount of work, but it could be done.

[noamross]

It appears that it actually occurs in download.file() by default with method = "libcurl". From ?download.file in 4.5.0:

When method "libcurl" is used, there is support for simultaneous downloads, so url and destfile can be character vectors of the same length greater than one (but the method has to be specified explicitly and not via "auto"). For a single URL and quiet = FALSE a progress bar is shown in interactive use.
...
When simultaneously downloading two or more files (see the url argument) and download of at least one file succeeds, 0 is returned with attribute retvals, which provides an integer vector of the same length as url with a result code for each file (0 for success and non-zero for failure).

In testing, all files appear on disk simultaneously on disk, though are finished at different times:

download.file(
  c("https://link.testfile.org/PDF100MB",
    "https://link.testfile.org/PDF100MB",
    "https://link.testfile.org/PDF100MB",
    "https://link.testfile.org/PDF100MB"),
  c("a.pdf", "b.pdf", "c.pdf", "d.pdf"),
  method = "libcurl"
)