googleapis / googleapis/librarian
sidekick: parallelize cargo semver-check for Rust
- Dominant language
- Go
- Stars
- 45
- Forks
- 61
- Avg merge
- 19h 11m
- Merged PRs (30d)
- 181
Description
`rust-bump-versions` and `rust-release` run all of the semver-checks in sequence. They need to because they all share the same `Cargo.lock`. We could take better advantage of our beefy Cloudtops and run some of this stuff in parallel.
Crazy idea to go fast:
- we have a "worker pool" with N workers (N = $(NCPU))
- for each package, we define a job:
- make a tmp dir
- sym link the files from the current directory to the tmp dir (the git files, we would not include the `target/` dir)
- copy the `Cargo.lock` (we don't want this linked)
- execute any `cargo` commands with `sccache`
- remove the tmp dir
- give the jobs to the worker pool, wait for them to finish
If crates that depend on each other must execute in sequence, it will take additional set up work, but I am not sure that is the case.
Contributor guide
Research direction
Start with the rust-bump-versions and rust-release entry points and trace how their semver checks currently run in sequence. Evaluate the proposed worker pool, temporary directories, linked files, copied Cargo.lock, and sccache usage, including whether dependent crates require ordering. Done means the semver checks safely run in parallel without conflicting Cargo.lock state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- build-system, release
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100