cockroachdb / cockroachdb/cockroach
kv: investigate dynamically escalating MVCC GC intent-resolution admission priority after repeated failures
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Summary**
MVCC GC resolves old/abandoned intents at a fixed admission-control priority — the cluster setting [`kv.gc.admission_priority`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/gc/gc.go#L142-L154), which defaults to `bulk_normal_pri`. On an overloaded node this work is exactly what AC deprioritizes, so GC-driven intent resolution gets starved while intents keep accumulating. Investigate whether the GC queue can **dynamically raise** this priority for a range when its intent resolution has failed repeatedly, so the system self-heals instead of requiring manual operator intervention.
**Motivation**
In a recent customer incident (support escalation), a sustained write hotspot drove unbounded intent buildup on a hot range. MVCC GC could not make progress resolving those intents because, at `bulk_normal_pri`, intent resolution was throttled by admission control on the CPU-saturated nodes.
**Observations / code references**
- GC intent cleanup runs at the static `gcAdmissionHeader` priority: [`mvcc_gc_queue.go:741-742`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L741-L742), built in [`gcAdmissionHeader`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L921-L923).
- Failure signals already exist: `GCResolveFailed` ([`:746`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L746)) and `GCTxnIntentsResolveFailed` ([`:759`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L759)). These could feed a per-range/per-store escalation decision.
- Background on why intent resolution is subject to AC and the priority model: [`intentresolver/admission.go`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/intentresolver/admission.go) (and the original overload regression #97108 that put intent resolution under AC).
**Questions to answer**
- [ ] Can the GC queue track consecutive intent-resolution failures (or throttling) per range and escalate the admission priority (e.g. `bulk_normal_pri` → `normal_pri` → `user_high_pri`) on retry?
- [ ] What's the right trigger — failure count, intent age/count, range CPU, or AC throttling signal specifically (vs. unrelated errors)?
**Next steps**
- [ ] Prototype per-range failure tracking + priority escalation in the GC queue.
- [ ] Evaluate against a reproduction of the hot-range intent-buildup scenario.
Epic: none
Jira issue: CRDB-64636
Contributor guide
Research direction
Start with pkg/kv/kvserver/mvcc_gc_queue.go, especially the GC intent cleanup around lines 741-759 and gcAdmissionHeader, then read pkg/kv/kvserver/intentresolver/admission.go. Determine whether repeated or admission-related failures can drive per-range priority escalation, and evaluate the result against the hot-range intent-buildup reproduction described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100