RFC: making sccache-dist work for OpenEmbedded/Yocto host-mode distributed builds
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.7k
- Forks
- 748
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 21
Description
I've been getting sccache-dist working for OpenEmbedded/Yocto host-mode
distributed builds, and the work has grown well past a single reviewable change.
I originally opened it as #2750, which I've now closed and am splitting into
small, focused PRs landed in waves. This issue is the discussion anchor for that
series, especially the scheduler work, before those PRs land.
What the series covers
Bug fixes (independent, landing first):
- A server-side panic when a finished compile reports neither exit code nor
signal. - Local fallback when a distributed compile fails or drops an output, instead of
failing the whole build. - gcc/rust distributed-compile correctness: force-local for feature probes and
PCH generation; ship the rust sysroot std, target spec, and rlibs; toolchain
packaging fixes for relocated and split-sysroot cross toolchains.
Scheduler and server reliability under high concurrency (a base-first stack):
- Prompt removal of stopped servers; topology-aware, deterministic routing;
idempotent job-state updates and lost/stale-job reaping; liveness-lease
reaping; restart-safe job accounting. - Per-toolchain preparation locks so concurrent compiles stop convoying, and
HTTP connection reuse.
Two compatibility surfaces I want to flag up front
- The liveness-lease reaper adds an
active_jobsfield to the heartbeat
request, which changes the bincode wire format: scheduler and build servers
must run matching binaries. Old-server to new-scheduler fails loudly;
new-server to old-scheduler silently drops the field. I'm happy to gate this
behind a protocol version if you'd prefer. - Connection reuse adds two
DistConfigkeys (withserde(default), so old
configs keep parsing) and a smallarc-swapdependency.
Questions for maintainers
- The scheduler routing work introduces some new policy surface (topology-aware
placement, a dynamically derived colocated de-weight, advertised core counts).
It is opt-in via config with defaults that preserve current behavior. Is that
direction welcome, or would you rather shape it before the PRs land? - There is a set of load-diagnosis instrumentation I used to find these issues.
I can submit it, fold the useful parts into the PRs that needed them, or drop
it. Preference?
I'll link the individual PRs here as they open. Happy to adjust the split,
ordering, or scope to match how you'd like to review this.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the scheduler and server-reliability work described in the issue, then review the listed compatibility surfaces and maintainer questions. The issue is a discussion anchor for a series of focused pull requests rather than a standalone change, so completion depends on an agreed split, ordering, and scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- build-system, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100