cockroachdb / cockroachdb/cockroach

kvserver: add telemetry metric for orphaned range-local keys found at startup

Open
#173,961 0 comments 0 reactions 1 assignee Claimed by @natakhot271 View on GitHub
A-kv-observability C-enhancement O-agent P-2 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
Orphaned range-local keys — created when a narrowing Raft snapshot fails to
clear the vacated span (the bug described in #73462, fixed on master by
`1ca06e49622`) — are completely silent in production builds. The startup invariant
check in `iterateRangeDescriptorsFromDiskHelper`
(`pkg/kv/kvserver/kvstorage/init.go`) is gated on `buildutil.CrdbTestBuild`; a
production binary simply skips over orphaned keys and starts normally. As a
result, we have no visibility into whether orphaned range-local keys exist in
customer clusters that have upgraded through pre-fix releases (v25.3.x and earlier).

This was surfaced by the investigation in #173559.

**Describe the solution you'd like**
In `iterateRangeDescriptorsFromDiskHelper`, count orphaned range-local keys (those
whose anchor falls outside every known range descriptor on the store) and report
the count via telemetry at the end of the startup scan:

```go
// In iterateRangeDescriptorsFromDiskHelper, where the current
// assertion/NextKey branch is:
orphanedRangeLocalKeyCount++
if orphanedRangeLocalKeyCount == 1 {
log.Warningf(ctx, "found orphaned range-local key %s outside of a known range ", key)
}

// At end of scan:
if orphanedRangeLocalKeyCount > 0 {
telemetry.Count("kv.store.orphaned_range_local_keys", orphanedRangeLocalKeyCount)
}
```

This gives us:
- A warning in logs on the first orphaned key seen (visible in support cases)
- A telemetry signal to understand how widespread the problem is across the fleet

**Describe alternatives you've considered**
1. Sentry reporting could be added later once we understand prevalence from telemetry.
2. A `metric.Counter` (Prometheus/DB Console) would be more appropriate if operators
needed to alert on this; telemetry is used here because the audience is Cockroach
Labs engineering — the counter is invisible to customers, quantized, and reset each
reporting period, which fits a one-time fleet-wide prevalence signal better than
continuous operational monitoring.

**Additional context**
The orphaned-key bug (#73462) is fixed in v25.4+ by `clearResidualDataOnNarrowSnapshot`
in `snapshot_apply_prepare.go`. Pre-fix binaries (v25.3.x and earlier) can leave
range-local keys (e.g. transaction records) stranded in the vacated span after a
narrowing snapshot. The keys are inert in production — no read/write path touches
them — but their presence is untracked. This metric would let us know whether
and how often this happens in real deployments before deciding whether a backport
or broader cleanup is warranted.

Epic: none

Jira issue: CRDB-67260

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.