cockroachdb / cockroachdb/cockroach

storage: rewrite CheckSSTConflicts

Open
#99,657 1 comment 0 reactions 0 assignees View on GitHub
A-storage C-cleanup T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The current implementation of the `storage.CheckSSTConflicts` function is very complicated, and it's very difficult to reason about its correctness. We've uncovered many bugs (eg, #99566) in its implementation as a result, and today largely rely on randomized testing (from KV nemesis and the unit test in #98408) to ensure correctness. We should rewrite and simplify it.

Some ideas:
* @itsbilal suggested a four-iterator approach. We'd use 1 range key iterator and 1 point iterator for both the engine and the sstable.
* We could implement a "tandem" iterator that keeps the engine iterator positioned appropriately as the sstable iterator moves across the keyspace. There are some subtleties in what "positioned appropriately" means (eg, the engine iterator may need to read _before_ the sstable iterator's position in order to observe a range key). Separating out the mechanics of keeping the two iterators in sync should make the actual conflict checking more tractable.

Jira issue: CRDB-26007

Contributor guide

Open the contributing guide

Research direction

Start with storage.CheckSSTConflicts, then read the KV nemesis coverage and the unit test from #98408. Review the failure described in #99566 and the proposed four-iterator or tandem-iterator approach. Done means a simpler implementation whose conflict-checking behavior remains correct under the existing randomized and unit tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.