googleapis / googleapis/librarian

sidekick: parallelize cargo semver-check for Rust

Open
#3,225 0 comments 0 reactions 0 assignees View on GitHub
sidekick
Dominant language
Go
Stars
45
Forks
61
Avg merge
19h 11m
Merged PRs (30d)
181

Description

`rust-bump-versions` and `rust-release` run all of the semver-checks in sequence. They need to because they all share the same `Cargo.lock`. We could take better advantage of our beefy Cloudtops and run some of this stuff in parallel.

Crazy idea to go fast:

- we have a "worker pool" with N workers (N = $(NCPU))
- for each package, we define a job:
- make a tmp dir
- sym link the files from the current directory to the tmp dir (the git files, we would not include the `target/` dir)
- copy the `Cargo.lock` (we don't want this linked)
- execute any `cargo` commands with `sccache`
- remove the tmp dir
- give the jobs to the worker pool, wait for them to finish

If crates that depend on each other must execute in sequence, it will take additional set up work, but I am not sure that is the case.

Contributor guide

Open the contributing guide

Research direction

Start with the rust-bump-versions and rust-release entry points and trace how their semver checks currently run in sequence. Evaluate the proposed worker pool, temporary directories, linked files, copied Cargo.lock, and sccache usage, including whether dependent crates require ordering. Done means the semver checks safely run in parallel without conflicting Cargo.lock state.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, release
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.