When crates.io gives 429, cargo should back off and retry later

Problem

Our workspace contains 46 cargo packages. (Because cargo insists that each crate must be a separate package, and we want to split up crates for code sanity and compilation time reasons.)

This means that in our recent release, our on-duty release technician hit the rate limit. This aborted publication of the workspace, requiring manual retries and wrangling.

Steps

Have a workspace with more than 30 (the current burst rate limit) crates. Try to publish it by publising each crate, in topo order, with cargo publish (using some automated tool).

Possible Solution(s)

cargo should handle a 429 response by backing off and retrying, using an exponential backoff algorithm.

In https://github.com/rust-lang/crates.io/issues/1643 the crates.io team report already having raised the rate limit. In the error message from crates.io they suggest emailing help@ to ask for a rate limit increase. Such a workflow is IMO undesirable, especially as Rust gets more adoption.

Notes

I don't think increasing the rate limit (globally, or on request) is the right fix. If 429 is a hard error there is a tension between preventing misuse, and not breaking large projects' releases. But this tension can be abolished by handling 429 gracefully.

#13397 would probably have assisted the recovery from this situation (and also the local disk space problem our releasae technician also ran into).

See also: https://github.com/rust-lang/crates.io/issues/3229 (requesting docs) #6714 (requesting better error message display).

Version

> cargo version --verbose                                                                                                                                                                                                                                                                             16:57:11
cargo 1.76.0 (c84b36747 2024-01-18)
release: 1.76.0
commit-hash: c84b367471a2db61d2c2c6aab605b14130b8a31b
commit-date: 2024-01-18
host: x86_64-unknown-linux-gnu
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.5.0-DEV (sys:0.4.70+curl-8.5.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Arch Linux Rolling Release [64-bit]

(edited to fix ticket links)

Start at the cargo publish entry point and trace how crates.io responses are handled, focusing on the 429 case. Review the issue discussion and related references before deciding the retry behavior. Done means cargo publish backs off and retries 429 responses using exponential backoff, with tests covering the behavior.