Coalesce sync flights when on-demand sync lands
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 4
- Forks
- 1
- Avg merge
- 3h 7m
- Merged PRs (30d)
- 523
Description
Problem
sync_project_files and sync_catalog serialize concurrent callers on a mutex. When on-demand project sync becomes reachable from the serving path, every waiter behind that mutex will issue its own conditional request to the upstream, so a burst of requests for one cold project becomes a burst of upstream requests rather than one.
This is not reachable today. #1302 established that waited is never true in production: sync_project_files has exactly one production caller, the catalog job, which iterates distinct projects and cannot overlap for one repository because the scheduler admits on a (kind, scope) conflict key; and sync_catalog's second caller is reachable only through peryx prefetch, which opens the store writable and so cannot run beside a serving process.
#1302 fixed the correctness half by deleting a shortcut that reported NotModified to a waiter without observing any upstream response. It deliberately did not build coalescing, because building it for callers that do not exist is speculative. This issue records the requirement for when they do.
What this needs, when it is needed
A waiter must learn the producer's outcome, not merely that the producer finished. Three outcomes are distinct and the current ProjectSyncOutcome collapses at least two of them: the producer published new content, the producer received a genuine 304, the producer failed.
The counting collision is the trap, and it is why this is not a small change. catalog_job.rs does report.changed += 1 per outcome. If waiters receive Published, one publication counts as N changes. Fixing that means either a distinct "another flight published this" outcome or moving the counting decision out of the per-caller path. Decide which before writing the flight, not after — this is the naming-collision class that forced PR #2089 to rename expired to timed_out, where a field's condition became reachable by a second cause and silently started meaning two things.
Also settle what a waiter observes when the producer is cancelled, which is a fourth case and has no representation today.
Acceptance criteria
- Concurrent callers for one project produce one upstream request, and each caller learns the producer's actual outcome.
- A publication counts as one change in catalog reporting regardless of how many callers waited on it. Assert the count, not just the outcome.
- A producer failure is reported as a failure to every waiter, never as a reuse of an existing generation.
- A cancelled producer has a defined observable outcome, pinned by a test.
ProjectSyncOutcome's documented meanings stay true for every path that can return each variant.
Boundary
Coalescing and outcome propagation for the sync flights. Do not open this until on-demand project sync from the serving path is actually reachable — until then there is no herd to prevent, and the mutex is sufficient. Note in the pull request which caller made it reachable, so the justification is on the record.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with sync_project_files, sync_catalog, and catalog_job.rs, then read the existing ProjectSyncOutcome semantics and the reasoning in #1302. Define how published, 304, failed, and cancelled producer outcomes reach every waiter before implementing coalescing. Done means one upstream request per concurrent project, one catalog change count per publication, and tests pinning failure, cancellation, and each outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100