cockroachdb / cockroachdb/cockroach

kv: investigate perturbation/full/partition recovery-throughput failures

Open
#173,562 1 comment 0 reactions 1 assignee Claimed by @stevendanna View on GitHub
A-testing C-bug C-test-failure O-roachtest P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

`perturbation/full/partition` fails frequently on master and release branches. This issue tracks the investigation; the auto-filed failure issues (#171399 on master, #171797/#172639/#173472 on release branches) will keep accruing individual occurrences.

## What the test checks

The partition perturbation isolates an entire region (4 of 12 nodes), removing ~1/3 of the leaseholders, then heals the partition. Because throughput is *expected* to collapse while the partition is active, the test skips the perturbation-interval gate and only gates the **recovery** interval: after the partition heals, foreground throughput must return to within 80% of the pre-perturbation baseline over the 5m recovery window. See `pkg/cmd/roachtest/tests/perturbation/network_partition.go`. Latency gates are disabled, so every failure is "throughput did not recover to baseline in time."

## Data (master, 36 failing runs, Jun–Aug 2026)

Max throughput impact ratio (baseline/measured) across failing runs:

- 20 runs in 1.30–1.99x
- 16 runs ≥ 2.0x, tail to 4.59x

The most recent failures (Aug 13–17) are the worst seen, clustered at 2.25–3.01x (recovery throughput ~35–45% of baseline), suggesting a possible regression in mid-August worth bisecting.

## Mitigation in place

To reduce noise on master while we investigate, the recovery gate for this test has been loosened from 1.25 to 2.0 (throughput may drop to 50% of baseline before failing). This does not silence the current August-class failures (all > 2.0), which we want to keep visible.

## Investigation directions

- Bisect the mid-August worsening.
- Determine where recovery time goes: lease re-acquisition, replication catch-up / snapshots, admission-control token drain, latch/lock contention.
- Use the recovery-window trace sampling added alongside the threshold bump (off by default; enable via `PERTURBATION_OVERRIDE`) to capture traces for slow requests during the recovery window and inspect where time is spent.

Epic: none

Jira issue: CRDB-66876

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.