cockroachdb / cockroachdb/cockroach
spanconfigkvsubscriber: replica of a GC'ed span does not respect SystemSpanConfigs
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In our test cluster we observed two ranges corresponding to dropped tables not respecting protection policies enforced by SystemSpanConfigs. The replicas of these empty ranges were still evaluating batch requests and causing them to fail with ` ‹ERROR: batch timestamp 1698960957.937065386,0 must be after replica GC threshold 1698967771.741796894,0 (SQLSTATE XXUUU)›`. The running theory is that with the following sequence of operations:
- Drop DB (t = 1)
- SCHEMA CHANGE GC Job writes range deletion at t=1, waits for span to be empty
- Write SystemSpanConfig PTS (t = 2) covering the dropped range
- MVCC GC Runs up to time 2
- Span is now empty, SCHEMA CHANGE GC JOB sees span is empty, deletes span configuration.
We end up in a situation where the `spanconfigstore` does not find any overlapping span configs for the empty range and so does not run the logic to check if any system span configs apply to that range - https://github.com/cockroachdb/cockroach/blob/master/pkg/spanconfig/spanconfigstore/store.go#L199. In this way it misses any protection policies that should hold up the GCThreshold and allows GC to move past the protected timestamp. We think it makes sense to apply a default zone config with the system span configs combined into it to such ranges. We are still attempting to reproduce this locally.
Jira issue: CRDB-33239
Contributor guide
Assessment
This issue has not been assessed yet.