cockroachdb / cockroachdb/cockroach
kv: last range in tenant may never get GC'd
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When a tenant's last range has no split point at the next tenant's start key, the range extends past the tenant boundary into the next tenant's keyspace. Any such tenant-straddling range is never MVCC-GC'd. See [repro](https://github.com/cockroachlabs/cockroach/pull/4405) .
A tenant's last range straddles the next tenant's boundary in two reachable ways:
1. System tenant: it never gets a split installed at /Tenant/2, so when the lowest secondary tenant ID is ≥ 3 its last range runs [/Table/, /Tenant/K).
2. Secondary tenant: after dropping an intermediate tenant, the lower tenant's tail range merges across the now-empty gap (once the tenant-boundary split's ~1h sticky bit from tenant creation expires) up to the next surviving tenant's prefix. For example: consider tenants 7, 8, 9; tenant 8 is dropped; last range in tenant 7 is no longer gc'able.
**Claude generated explanation on why this range never gets gc'd**
Before deciding whether a range can be garbage-collected, the MVCC GC code runs a protected-timestamp check over the range's own descriptor span. For a range whose start and end keys belong to two different tenants, that check errors out, and the error is treated as "don't GC this range" — so the range is never collected.
1. mvccGCQueue.shouldQueue and mvccGCQueue.process both call repl.checkProtectedTimestampsForGC(...) — pkg/kv/kvserver/mvcc_gc_queue.go:271 and :690.
2. checkProtectedTimestampsForGC calls readProtectedTimestampsRLocked — pkg/kv/kvserver/replica_protected_timestamp.go:120.
3. readProtectedTimestampsRLocked builds a span from the range's own descriptor, [desc.StartKey, desc.EndKey), and reads protection timestamps over it — pkg/kv/kvserver/replica_protected_timestamp.go:66-72.
4. That read routes through the PTS reader adapter (pkg/spanconfig/spanconfigptsreader/adapter.go:49) → KVSubscriber.GetProtectionTimestamps (pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:388), which calls
Store.ForEachOverlappingSpanConfig (kvsubscriber.go:394).
5. Store.ForEachOverlappingSpanConfig decodes the tenant from the span's start and end keys and, if they differ (beyond a one-tenant tolerance at :212-216), returns span ... crosses tenant boundary —
pkg/spanconfig/spanconfigstore/store.go:217-220.
That error propagates back up: checkProtectedTimestampsForGC returns it (replica_protected_timestamp.go:121-123); shouldQueue turns any error into (false, 0), i.e. never enqueue (mvcc_gc_queue.go:272-275); and process returns (false, err),
i.e. never process (mvcc_gc_queue.go:691-693).
Jira issue: CRDB-67900
Contributor guide
Assessment
This issue has not been assessed yet.