solver: ReleaseUnreferenced can hold the cache.db write lock for extended periods under sustained load

What happens

cacheManager.ReleaseUnreferenced is currently invoked from two places:

• After every build, through throttledReleaseUnreferenced at control/control.go:715, with a hard-coded 5-minute throttle window.
• After every Prune, as a fallback in the defer block at control/control.go:255-263 whenever the prune produced any records.

The execution path is:

1. cacheManager.ReleaseUnreferenced (solver/cachemanager.go:45) walks the entire _links bucket and calls Store.Release for every orphan resultID.
2. Store.Release (solver/bboltcachestorage/storage.go:200) runs emptyBranchWithParents (storage.go:249) recursively inside a single db.Update write transaction.
3. The recursion visits the _result, _links, _byresult, and _backlinks buckets in turn, and the same write transaction keeps bbolt's exclusive write lock until the entire subgraph has been cleared.

Why it matters

Under sustained parallel build load we've observed the following, all reproducible in principle:

• The _links bucket grows roughly linearly with build volume, and both the recursion depth and branching factor in emptyBranchWithParents grow with it.
• A single ReleaseUnreferenced invocation can hold the cache.db write transaction for seconds, and in the worst case tens of seconds, depending on the orphan population at that moment.
• Anything else that goes through db.Update on the same store (AddResult, AddLink, or the next Release) queues behind bbolt's single writer lock during that window.
• As a result, solver-hot-path operations (Query, Load, AddResult, AddLink) show a visible tail-latency increase while a release pass is in flight.

The current throttling strategy is hard-coded at control/control.go:145:

c.throttledReleaseUnreferenced = throttle.After(5*time.Minute, func() {
    c.releaseUnreferencedCache(context.TODO())
})

In a high-concurrency workload this 5-minute constant has a few practical limitations:

• Not configurable. Every deployment shares the same constant regardless of build density, so operators cannot tune it against their own load profile.
• No way to trigger manually. Even when an operator knows the daemon is in a lull and would like to run a release pass right now, there is no entry point to do so.
• Backlog grows faster than the release pass can drain it. Under sustained traffic, the number of new orphan entries produced within a 5-minute window can exceed what a single pass clears. The release cadence falls behind orphan production, and each subsequent pass has more work to do — so the pass gets slower over time rather than reaching steady state.

Related prior work

This isn't an isolated issue; the community has been chipping away at "cache.db index correctness + physical space reclamation" through several patches:

• #4353 (cache: fix cache leak) — cleans up compression-variant leases when snapshot load fails at init, preventing blob leaks.
• #5116 (bboltcachestorage: only delete link after releasing result) — fixes emptyBranchWithParents so it no longer deletes subLinks when subResult is still non-empty, keeping Release semantics correct.
• #7135 (cache/metadata: remove obsolete indexes when replacing values) — prevents stale _index entries from being left behind when a record's indexed value is replaced.
• #7138 (opt-in metadata database compaction, in review) — introduces opt-in bbolt physical compaction for cache.db, history.db, metadata_v2.db, and containerdmeta.db.

Those four cover, respectively, "is Release correct?", "do we still produce new orphan index entries?", and "how do we reclaim the physical space bbolt has already given up on?". The remaining question — when ReleaseUnreferenced runs, at what cadence, and whether an operator can drive it explicitly — isn't addressed by any of them.

Proposed direction

We'd like to confirm the direction at the issue level first, and are happy to submit a PR if the maintainers agree. The minimal, non-invasive shape we have in mind is two complementary changes:

1. A debug endpoint to trigger ReleaseUnreferenced manually, for example POST /debug/cache/release-unreferenced, aligned in style with the /debug/compaction endpoint added by #7138. This lets an operator drive a release pass at a known-quiet moment instead of waiting for the post-build throttle to expire.
2. Lift the 5-minute constant into buildkitd.toml, e.g. [cache] releaseUnreferencedInterval, keeping the default at 5m so existing deployments are unaffected.

Explicitly out of scope for this issue and any follow-up PR, to keep the change surface small:

• Splitting the transaction granularity inside Release (turning the single db.Update recursion into batched sub-transactions). That's an independent performance improvement and worth its own PR.
• Changing the recursion in emptyBranchWithParents. That's a correctness-adjacent area and, again, better on its own.
• Anything that touches #7138's compaction scheduler. The release cadence and physical compaction are independent concerns; we'd rather not couple them.

Reproduction

We're preparing a minimal reproducer along these lines and will add it before opening the PR:

• Run buildkitd with the oci worker.
• Drive it with a persistent loop that exports cache on every build, e.g. ~20 goroutines each running buildctl build ... --export-cache=type=local,dest=....
• Sample the _links bucket size every 30s (via bbolt stats or by instrumenting bbolt.Bucket.Stats()).
• Once _links grows into the tens-of-thousands range, instrument the db.Update duration of the next post-build ReleaseUnreferenced.
• Expected observation: the db.Update hold time increases monotonically with _links size, and external calls that need db.Update (subsequent build steps, buildctl du, etc.) see correlated latency increases during the release pass.

Environment

• BuildKit: master, base commit 99bd9de47d29269020476c3eea5898f6038fa0a1.
• Runtime: multi-replica buildkitd behind a consistent-hash router.
• Worker: runc + overlayfs.
• Steady state: cache.db in the multi-GiB range, sustained parallel builds; exact numbers omitted, but the qualitative pattern (release pass lengthens over time, tail latency correlates with release duration) is stable across the fleet.

What we'd like from maintainers

• Confirm or reject the "manual trigger + configurable throttle" direction.
• If accepted: guidance on the preferred surface — HTTP debug endpoint (POST /debug/cache/release-unreferenced) vs. a buildctl debug subcommand vs. both.
• If there's a different direction you'd prefer (for example, replacing the throttle with a dynamic threshold based on orphan count or _links growth rate), we're happy to rework along those lines.

Happy to submit a PR once the direction is confirmed.

Start with control/control.go:145, 255-263, and 715, then read solver/cachemanager.go:45 and bboltcachestorage/storage.go:200 and 249 to understand the release path and its transaction scope. Compare the proposed debug trigger with the /debug/compaction endpoint from #7138. Done means maintainers' preferred trigger surface and configurable interval are implemented with the existing 5-minute default, without changing release transaction granularity or compaction scheduling.