Download update packages concurrently before running the bulk upgrader

Start in wp-cli/wp-cli with WpHttpCacheManager and its existing cache validation, then trace extension-command's update_many() through the whitelist loop and Plugin_Upgrader::bulk_upgrade(). Review the existing cache and updater tests, including the Behat HTTP mock behavior. Done means supported pending packages are prefetched safely, failures fall back to serial downloads, installs remain unchanged, and normal and machine-readable output stay correct.

Feature request

I recently improved the bulk plugin updater speed within Minn Admin and thought I would backport the idea to WP-CLI.

wp plugin update --all (and wp theme update --all) hands the whole list to Plugin_Upgrader::bulk_upgrade(), which downloads and installs each package one after the other. The downloads are the part that does not need to be serial. The install step has to stay serial (shared database, object cache, maintenance mode), but nothing about fetching zips from downloads.wordpress.org requires waiting for the previous one to finish.

Proposal: before calling bulk_upgrade(), fetch every pending package side by side into WP-CLI's existing HTTP cache. The upgrader then finds each package already on disk, exactly the way it does today on a warm cache, and the install phase runs unchanged.

Why this fits the cache manager

update_many() already whitelists every package URL with WpHttpCacheManager, and the cache manager already answers the upgrader's download_url() calls from disk through its pre_http_request filter when the file is cached. So the only new piece is filling that cache concurrently ahead of time. No new hook into the upgrader, no change to how a package is validated or installed, and a package whose prefetch fails is simply downloaded by the upgrader as before.

Concretely:

• wp-cli/wp-cli: a WpHttpCacheManager::prefetch( array $urls, $concurrency = 6 ) method. For URLs that are whitelisted and not yet cached, it runs Requests::request_multiple() in chunks of six into temp files, runs the existing validate_downloaded_file() check (from wp-cli/wp-cli#6149), and imports the good ones into the FileCache. It goes through the http_request_options hook (so the Behat HTTP mock is honored: its request_multiple() throws, the method catches, and the serial path runs), and it does nothing when the cache is disabled, when fewer than two packages are pending, or when WP_HTTP_BLOCK_EXTERNAL / WP_PROXY_HOST is set, since those are enforced by the WordPress HTTP API which the prefetch does not go through. For the same reason WordPress's http_request_args / pre_http_request filters are not applied to the prefetch, so a package that depends on them (an authorization header added by an update plugin, say) fails validation there and is downloaded by the upgrader exactly as today.
• wp-cli/extension-command: one call in update_many() after the whitelist loop, guarded by method_exists() so the package keeps working on a WP-CLI without the method.

Output gains one line, Downloading N packages..., before the per-item Updating X... block. Under --format=json|csv the quiet logger is already in place at that point, so machine output is unchanged.

Measurements

Local Cove site, WordPress 7.1.1, 11 wordpress.org plugin updates pending (23.9 MB of packages), the plugins directory restored from a tarball between rounds, WP-CLI cache directory emptied before each cold round. Wall time of wp plugin update --all, same code and same site, the only variable being when the network happened to be fast or slow:

The saving is the serial sum of the transfers minus the slowest batch, so it tracks how slow the path to downloads.wordpress.org is at that moment. In the very fast window the download phase went from 1.6 to 1.8s down to 0.6 to 0.7s (Prefetched 11 of 11 packages in 0.6s under --debug=http), which is about a second of wall time and inside run-to-run noise (one of the three pairs came out the other way). Warm cache is unchanged (all packages already on disk, nothing to prefetch; 7.1s vs 6.7s in the same session). Per-phase timing from upgrader_* hooks in the fast window: 4.2s downloading, 1.0s unzipping, 5.0s in install_package(); the prefetch takes the download phase from 4.2s to 1.8s.

The remaining 5.0s install phase is mostly not I/O: WordPress core's move_dir() has an unconditional usleep( 200000 ) after every successful rename (added in 6.2 for VirtualBox shared folders), and the upgrader calls move_dir() twice per plugin since 6.3 (temp backup, then install), so a bulk update of N items sleeps 0.4·N seconds. That is a core ticket, filed separately; it is mentioned here because it is the other half of the same slowness and because the two fixes are additive (with both, the same run takes 4.6s in the fast window).

Prior art

Searched the wp-cli org (all three repos plus wp-cli/ideas), WordPress/wordpress-develop, WordPress Trac and GitHub code search before writing this; nothing proposes or implements it.

• #29 (2017) asked for asynchronous plugin installation by running several wp plugin install processes at once, closed because two processes share the database and object cache. This proposal keeps a single process and a serial install; only the network transfer overlaps.
• #444 / #501 (opcache invalidation after plugin update) is the only other thread that looks at what happens inside the install phase; it is adjacent, not overlapping.
• Composer made parallel package downloads the headline of its 2.0 release (2020); the same shape (fetch everything, then install serially) is what is proposed here.
• The WordPress admin bulk updater and Minn Admin's "Update everything" have the same serial-download shape; Minn Admin prefetches with Requests::request_multiple() and hands the files to the upgrader through upgrader_pre_download, which is where the numbers above were first observed (a wp-admin bulk update of the same 11 plugins ran 21 to 33s; the prefetching panel 12.4s). Reusing WP-CLI's cache manager instead of that filter is cleaner here because the cache path already exists and is already tested.

Questions for the maintainers

1. Is the cache manager the right home for prefetch(), or would you rather keep the change inside extension-command (it would then have to duplicate the cache key format and the download validation)?
2. Concurrency of six is the number the Minn Admin implementation settled on as gentle on shared hosts; happy to make it a config value or an environment variable if you want it tunable.
3. Should plugin install a b c get the same treatment in a follow-up? It needs the package URLs first (one plugins_api() call per slug, also serial today), so it is a separate change.

Branches with tests are ready for both repositories; opening this first per the contributing guide.