Cluster-wide compile coordination

The problem

Steady-state isn't where this hurts. On the slow hardware sccache shines on, a project that takes 10+ hours from cold can take ~1 hour with a warm cache. At a typical 5–10 commits/hour the storage cache absorbs most of the work and runners don't overlap on the same compiles much.

The problem is the cache-reset wavefront. When something invalidates a large chunk of cache keys at once (a central header change, a toolchain bump, a library upgrade) every runner that hits that commit suddenly has a long tail of uncached compiles in front of it. And because the keys are content-addressed, every runner's tail is mostly the same set of keys (when their actual commits don't change those derived hashes).

The case I'm seeing is:

• The first runner hits the reset and rebuilds most everything from scratch. It takes hours to slowly go through all the build steps.
• A second runner joins 10 minutes later. The first few cached steps land cleanly (objects the first runner has already finished and published), but then it hits the wavefront (keys the first runner hasn't reached yet) and starts compiling them itself.
• The third, fourth, fifth runners do the same. All of them grind through the same wavefront in parallel, producing the same artifacts.

Multiply by the size of the fleet and the depth of the wavefront and that's where the wasted CPU lives, and so the opportunity for better usage of that CPU power.

Why storage caching alone can't fix it

Storage solves the temporal dedup ("the second build can reuse the first"), but it doesn't solve the concurrent dedup the wavefront needs: "the second runner shouldn't have to wait for the first to finish and put, but it also shouldn't redo the work in the meantime."

Some form of coordination seems like the missing primitive. At minimum, "is anyone else already compiling this hash right now, and should I do something in the meantime?". That cooperation would be an optimization hint, not a correctness primitive: content-addressed storage stays the source of truth, and a bug in coordination just degrades to today's behaviour (a redundant compile).

What I'd like to ask

Before going further on the design, I'd like to check whether this is a direction you'd be open to in sccache itself:

• Is cluster-wide compile coordination in scope for sccache, or do you see it living as a separate layer?
• If in scope, what shape would you want it to take? (A pluggable backend like the storage chain? A specific backend you'd prefer? A "coordinate via the same backend the cache is using" pattern?)
• What would change your mind from "interesting, but not for sccache" to "yes, please open a PR"?

Example implementation

For context: I do have a working prototype with a Redis-based backend (lease + pubsub for waiter wakeup), running in production on a riscv64 PyTorch CI fleet. On one machine in a single trunk build it deduped 716 of 910 wavefront compiles (78%) with total of 10 machines building different commits, which is roughly the wavefront pattern playing out in numbers. I'm happy to share the design and code if you'd like to see it, but I didn't want to lead with the implementation if the answer to the first question is "we don't think this belongs here."

The issue does not name project files, tests, or entry points. Start by reviewing the proposed coordination model and the linked Redis prototype, then check whether maintainers want this in sccache or a separate layer. Done would require an agreed scope and design before implementation work can be defined.