cockroachdb / cockroachdb/cockroach

kv: investigate dynamically escalating MVCC GC intent-resolution admission priority after repeated failures

Open
#171,517 0 comments 0 reactions 0 assignees View on GitHub
A-admission-control A-kv C-investigation O-agent O-postmortem O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Summary**

MVCC GC resolves old/abandoned intents at a fixed admission-control priority — the cluster setting [`kv.gc.admission_priority`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/gc/gc.go#L142-L154), which defaults to `bulk_normal_pri`. On an overloaded node this work is exactly what AC deprioritizes, so GC-driven intent resolution gets starved while intents keep accumulating. Investigate whether the GC queue can **dynamically raise** this priority for a range when its intent resolution has failed repeatedly, so the system self-heals instead of requiring manual operator intervention.

**Motivation**

In a recent customer incident (support escalation), a sustained write hotspot drove unbounded intent buildup on a hot range. MVCC GC could not make progress resolving those intents because, at `bulk_normal_pri`, intent resolution was throttled by admission control on the CPU-saturated nodes.

**Observations / code references**

- GC intent cleanup runs at the static `gcAdmissionHeader` priority: [`mvcc_gc_queue.go:741-742`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L741-L742), built in [`gcAdmissionHeader`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L921-L923).
- Failure signals already exist: `GCResolveFailed` ([`:746`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L746)) and `GCTxnIntentsResolveFailed` ([`:759`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/mvcc_gc_queue.go#L759)). These could feed a per-range/per-store escalation decision.
- Background on why intent resolution is subject to AC and the priority model: [`intentresolver/admission.go`](https://github.com/cockroachdb/cockroach/blob/bfb479b7e356f7fa1ff5d43257b9cf8d2a63debd/pkg/kv/kvserver/intentresolver/admission.go) (and the original overload regression #97108 that put intent resolution under AC).

**Questions to answer**

- [ ] Can the GC queue track consecutive intent-resolution failures (or throttling) per range and escalate the admission priority (e.g. `bulk_normal_pri` → `normal_pri` → `user_high_pri`) on retry?
- [ ] What's the right trigger — failure count, intent age/count, range CPU, or AC throttling signal specifically (vs. unrelated errors)?

**Next steps**
- [ ] Prototype per-range failure tracking + priority escalation in the GC queue.
- [ ] Evaluate against a reproduction of the hot-range intent-buildup scenario.

Epic: none

Jira issue: CRDB-64636

Contributor guide

Open the contributing guide

Research direction

Start with pkg/kv/kvserver/mvcc_gc_queue.go, especially the GC intent cleanup around lines 741-759 and gcAdmissionHeader, then read pkg/kv/kvserver/intentresolver/admission.go. Determine whether repeated or admission-related failures can drive per-range priority escalation, and evaluate the result against the hot-range intent-buildup reproduction described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.