cockroachdb / cockroachdb/cockroach
kvserver: add telemetry metric for orphaned range-local keys found at startup
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Orphaned range-local keys — created when a narrowing Raft snapshot fails to
clear the vacated span (the bug described in #73462, fixed on master by
`1ca06e49622`) — are completely silent in production builds. The startup invariant
check in `iterateRangeDescriptorsFromDiskHelper`
(`pkg/kv/kvserver/kvstorage/init.go`) is gated on `buildutil.CrdbTestBuild`; a
production binary simply skips over orphaned keys and starts normally. As a
result, we have no visibility into whether orphaned range-local keys exist in
customer clusters that have upgraded through pre-fix releases (v25.3.x and earlier).
This was surfaced by the investigation in #173559.
**Describe the solution you'd like**
In `iterateRangeDescriptorsFromDiskHelper`, count orphaned range-local keys (those
whose anchor falls outside every known range descriptor on the store) and report
the count via telemetry at the end of the startup scan:
```go
// In iterateRangeDescriptorsFromDiskHelper, where the current
// assertion/NextKey branch is:
orphanedRangeLocalKeyCount++
if orphanedRangeLocalKeyCount == 1 {
log.Warningf(ctx, "found orphaned range-local key %s outside of a known range ", key)
}
// At end of scan:
if orphanedRangeLocalKeyCount > 0 {
telemetry.Count("kv.store.orphaned_range_local_keys", orphanedRangeLocalKeyCount)
}
```
This gives us:
- A warning in logs on the first orphaned key seen (visible in support cases)
- A telemetry signal to understand how widespread the problem is across the fleet
**Describe alternatives you've considered**
1. Sentry reporting could be added later once we understand prevalence from telemetry.
2. A `metric.Counter` (Prometheus/DB Console) would be more appropriate if operators
needed to alert on this; telemetry is used here because the audience is Cockroach
Labs engineering — the counter is invisible to customers, quantized, and reset each
reporting period, which fits a one-time fleet-wide prevalence signal better than
continuous operational monitoring.
**Additional context**
The orphaned-key bug (#73462) is fixed in v25.4+ by `clearResidualDataOnNarrowSnapshot`
in `snapshot_apply_prepare.go`. Pre-fix binaries (v25.3.x and earlier) can leave
range-local keys (e.g. transaction records) stranded in the vacated span after a
narrowing snapshot. The keys are inert in production — no read/write path touches
them — but their presence is untracked. This metric would let us know whether
and how often this happens in real deployments before deciding whether a backport
or broader cleanup is warranted.
Epic: none
Jira issue: CRDB-67260
Contributor guide
Assessment
This issue has not been assessed yet.