cockroachdb / cockroachdb/cockroach
kvserver: possible to merge away liveness range during bootstrap
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
It's possible to merge away the liveness range temporarily before re-splitting it, and only during cluster bootstrap. We've only seen this happen in unit tests with 50ms replica scan intervals, and previously thought to have been fixed in #75939.
Right during cluster bootstrap before the span config reconciler has ever had a chance to run (i.e. system.span_configurations is empty), it's possible that the subscriber is subscribed to an empty span config state. So it was not been meaningfully "updated" in any sense of the word, but we still previously set a non-empty last-updated timestamp, something various components in KV rely on as proof that we have span configs as of some timestamp. As a result, we saw KV incorrectly merge away the liveness range into adjacent ranges, and then later split it off once there's non-empty span config state. This bug existed as far back as 22.2. We don't think we've ever seen this happen outside of tests as up until recently, it would have instantly triggered the following fatal in the raftScheduler, which wants to prioritize the liveness range above all else:
panic: priority range ID already set: old=2, new=61, first set at ...
There was an earlier attempt in #100210 to fix this bug above by erroring out until a span config snapshot is available. That made it so that tests now needed to actively wait for a span config snapshot before relocating ranges manually or using certain kv queues. Adding that synchronization made lots of tests a whole lot slower (by 3+s each) despite reducing the closed timestamp interval, etc. These tests weren't really being harmed by the bug (== empty span config snapshot). So it's not clear that the bug fix is worth fixing.
Jira issue: CRDB-28403
Contributor guide
Assessment
This issue has not been assessed yet.