cockroachdb / cockroachdb/cockroach

spanconfigkvsubscriber: replica of a GC'ed span does not respect SystemSpanConfigs

Open
#113,867 3 comments 0 reactions 0 assignees View on GitHub
A-kv branch-master C-bug O-testcluster P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In our test cluster we observed two ranges corresponding to dropped tables not respecting protection policies enforced by SystemSpanConfigs. The replicas of these empty ranges were still evaluating batch requests and causing them to fail with ` ‹ERROR: batch timestamp 1698960957.937065386,0 must be after replica GC threshold 1698967771.741796894,0 (SQLSTATE XXUUU)›`. The running theory is that with the following sequence of operations:

- Drop DB (t = 1)
- SCHEMA CHANGE GC Job writes range deletion at t=1, waits for span to be empty
- Write SystemSpanConfig PTS (t = 2) covering the dropped range
- MVCC GC Runs up to time 2
- Span is now empty, SCHEMA CHANGE GC JOB sees span is empty, deletes span configuration.

We end up in a situation where the `spanconfigstore` does not find any overlapping span configs for the empty range and so does not run the logic to check if any system span configs apply to that range - https://github.com/cockroachdb/cockroach/blob/master/pkg/spanconfig/spanconfigstore/store.go#L199. In this way it misses any protection policies that should hold up the GCThreshold and allows GC to move past the protected timestamp. We think it makes sense to apply a default zone config with the system span configs combined into it to such ranges. We are still attempting to reproduce this locally.

Jira issue: CRDB-33239

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.