cockroachdb / cockroachdb/cockroach
kv: investigate perturbation/full/partition recovery-throughput failures
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
`perturbation/full/partition` fails frequently on master and release branches. This issue tracks the investigation; the auto-filed failure issues (#171399 on master, #171797/#172639/#173472 on release branches) will keep accruing individual occurrences.
## What the test checks
The partition perturbation isolates an entire region (4 of 12 nodes), removing ~1/3 of the leaseholders, then heals the partition. Because throughput is *expected* to collapse while the partition is active, the test skips the perturbation-interval gate and only gates the **recovery** interval: after the partition heals, foreground throughput must return to within 80% of the pre-perturbation baseline over the 5m recovery window. See `pkg/cmd/roachtest/tests/perturbation/network_partition.go`. Latency gates are disabled, so every failure is "throughput did not recover to baseline in time."
## Data (master, 36 failing runs, Jun–Aug 2026)
Max throughput impact ratio (baseline/measured) across failing runs:
- 20 runs in 1.30–1.99x
- 16 runs ≥ 2.0x, tail to 4.59x
The most recent failures (Aug 13–17) are the worst seen, clustered at 2.25–3.01x (recovery throughput ~35–45% of baseline), suggesting a possible regression in mid-August worth bisecting.
## Mitigation in place
To reduce noise on master while we investigate, the recovery gate for this test has been loosened from 1.25 to 2.0 (throughput may drop to 50% of baseline before failing). This does not silence the current August-class failures (all > 2.0), which we want to keep visible.
## Investigation directions
- Bisect the mid-August worsening.
- Determine where recovery time goes: lease re-acquisition, replication catch-up / snapshots, admission-control token drain, latch/lock contention.
- Use the recovery-window trace sampling added alongside the threshold bump (off by default; enable via `PERTURBATION_OVERRIDE`) to capture traces for slow requests during the recovery window and inspect where time is spent.
Epic: none
Jira issue: CRDB-66876
Contributor guide
Assessment
This issue has not been assessed yet.