cockroachdb / cockroachdb/cockroach

roachtest: enforce deterministic benchmark cluster specs to prevent performance regression noise

Open
#168,520 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-perf C-enhancement O-agent T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A recent regression was introduced in kv benchmarks by adding `spec.RandomizeVolumeType()` and `spec.RandomlyUseXfs()` to the cluster spec (see [kv.go#L353](https://github.com/cockroachdb/cockroach/blob/90ba43f495776ddd8834579e31cf84f719be76be/pkg/cmd/roachtest/tests/kv.go#L353)). This causes nightly benchmark runs to use different storage configurations across runs, making performance comparisons unreliable. Similar regressions have occurred in the past (e.g., changing default machine types).

## Proposal

Introduce a "benchmark contract" mechanism that enforces deterministic hardware configurations for performance-tracked benchmarks:

1. **Declarative benchmark spec**: Define a struct or configuration that pins the full cluster spec for each benchmark (machine type, CPU, disk type, filesystem, number of disks, RAID config, cloud provider).

2. **CI lint/test enforcement**: Add a lint check or test that asserts benchmark-tagged tests (`Benchmark: true` in `TestSpec`) never use randomized spec options (e.g., `RandomizeVolumeType`, `RandomlyUseXfs`). This catches the class of bugs where "harmless" randomization or default changes silently break benchmark stability.

3. **Spec recording in results**: Record the full cluster spec alongside benchmark results so that configuration drift can be detected retroactively and discontinuities in performance timelines can be identified.

## Immediate fix

Remove `RandomizeVolumeType()` and `RandomlyUseXfs()` from nightly perf-tracked KV benchmarks. Randomization should be confined to weekly or dedicated correctness/robustness test variants.

## References

- Regression introduced at: [kv.go#L353](https://github.com/cockroachdb/cockroach/blob/90ba43f495776ddd8834579e31cf84f719be76be/pkg/cmd/roachtest/tests/kv.go#L353)
- `TestSpec.Benchmark` marker already exists in the registry and can be leveraged for enforcement.

Epic: none

Jira issue: CRDB-62985

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.