mozilla / mozilla/sccache

RFC: making sccache-dist work for OpenEmbedded/Yocto host-mode distributed builds

Open
#2,775 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
7.7k
Forks
748
Avg merge
4d 7h
Merged PRs (30d)
21

Description

I've been getting sccache-dist working for OpenEmbedded/Yocto host-mode
distributed builds, and the work has grown well past a single reviewable change.
I originally opened it as #2750, which I've now closed and am splitting into
small, focused PRs landed in waves. This issue is the discussion anchor for that
series, especially the scheduler work, before those PRs land.

What the series covers

Bug fixes (independent, landing first):

  • A server-side panic when a finished compile reports neither exit code nor
    signal.
  • Local fallback when a distributed compile fails or drops an output, instead of
    failing the whole build.
  • gcc/rust distributed-compile correctness: force-local for feature probes and
    PCH generation; ship the rust sysroot std, target spec, and rlibs; toolchain
    packaging fixes for relocated and split-sysroot cross toolchains.

Scheduler and server reliability under high concurrency (a base-first stack):

  • Prompt removal of stopped servers; topology-aware, deterministic routing;
    idempotent job-state updates and lost/stale-job reaping; liveness-lease
    reaping; restart-safe job accounting.
  • Per-toolchain preparation locks so concurrent compiles stop convoying, and
    HTTP connection reuse.

Two compatibility surfaces I want to flag up front

  1. The liveness-lease reaper adds an active_jobs field to the heartbeat
    request, which changes the bincode wire format: scheduler and build servers
    must run matching binaries. Old-server to new-scheduler fails loudly;
    new-server to old-scheduler silently drops the field. I'm happy to gate this
    behind a protocol version if you'd prefer.
  2. Connection reuse adds two DistConfig keys (with serde(default), so old
    configs keep parsing) and a small arc-swap dependency.

Questions for maintainers

  • The scheduler routing work introduces some new policy surface (topology-aware
    placement, a dynamically derived colocated de-weight, advertised core counts).
    It is opt-in via config with defaults that preserve current behavior. Is that
    direction welcome, or would you rather shape it before the PRs land?
  • There is a set of load-diagnosis instrumentation I used to find these issues.
    I can submit it, fold the useful parts into the PRs that needed them, or drop
    it. Preference?

I'll link the individual PRs here as they open. Happy to adjust the split,
ordering, or scope to match how you'd like to review this.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the scheduler and server-reliability work described in the issue, then review the listed compatibility surfaces and maintainer questions. The issue is a discussion anchor for a series of focused pull requests rather than a standalone change, so completion depends on an agreed split, ordering, and scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.