cockroachdb / cockroachdb/cockroach
kvserver: use WaitForApplication instead of splitDelayHelper
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The split delay helper solves a critical problem that can occur during rapid, cascading splits in CockroachDB, particularly when splits happen in ascending order (where each split divides the right-hand side of the previous split).
Each split creates a new replica. The leaseholder usually creates the right hand side leaseholder replica first, so it's often the case that this replica's MsgApp to other stores gets dropped (since they haven't split off the right hand side yet). This causes the leaseholder RawNode to wait and retry only after a couple of raft ticks (~100s of ms).
In the meantime, additional splits of the right hand side can occur. The leaseholder of the right hand side (r2) can't commit these until a quorum with followers has been reached, but as long as a quorum is "active" on the rightmost range, another split will succeed. This means an individual slow store may fall arbitrarily behind on the splits.
It may then get cut off from the raft log of the first node due to a size-based raft log truncation. All of the ranges will thus need a snapshot. However, the snapshots have an ordering constraint: they need to apply in left-to-right order, since the original unsplit range (r1) will occupy the entire keyspace that would be covered by the final split ranges.
The snapshot queue, however, is not aware of this ordering constraint. It will thus take potentially a very long time (and many failed attempts) until the snapshots have been processed. In the worst case, for 100 splits, each pass through all the replicas will only result in one successful snapshot (if, say, the queue always attempts to send snapshots in descending rangeID order). So it takes ~thousands of scanner cycles.
The same results if a follower needs a snapshot at the time of the first split: after the split, you have two followers that need a snapshot (the shrunk LHS and the new RHS).
The split delay helper waits for - or attempts to wait for - the right-hand side replicas to be properly initialized before allowing the next split to proceed. This ensures that each split is fully replicated before the next one begins, preventing the cascade of delays and snapshot issues.
However, it has a 45s max delay. Depending on the configured max snapshot rate and range size, this may not be enough. We've seen during scale testing that we would easily enter this scenario during a large IMPORT. Even though we didn't fully RCA it, it seems that this mechanism could be improved.
**Suggestion**
We remove the splitDelayHelper and replace it with a call to `waitForApplication`. Essentially, we wait for all live followers to have applied up to at least the local replica's log position.
It seems more appealing to use waitForApplication (vs. making splitDelayHelper more aggressive about waiting) since splitDelayHelper is fairly invasive in the sense that it interprets various bits of raft state in an attempt to understand the state of each replica. This kind of complexity is a bit unwieldy, and there are concerns about its robustness that could cause issues (block desirable splits) should it be allowed to delay indefinitely.
Waiting for "a command" to show up on all "vanilla live" followers seems much easier to rationalize.
I'll note that there already is a concept of "delayable" splits, and the splitDelayHelper only applies to those. The same would be true for this updated mechanism. Delayable splits are roughly manual splits, i.e. those not carried out by the split queue in response to size or load. In particular, splits issues from the SQL subsystem are delayable.
**Additional context**
What was the impact?
Add any other context about the problem here.
Jira issue: CRDB-48941
Contributor guide
Assessment
This issue has not been assessed yet.