cockroachdb / cockroachdb/cockroach
roachtest: enforce deterministic benchmark cluster specs to prevent performance regression noise
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A recent regression was introduced in kv benchmarks by adding `spec.RandomizeVolumeType()` and `spec.RandomlyUseXfs()` to the cluster spec (see [kv.go#L353](https://github.com/cockroachdb/cockroach/blob/90ba43f495776ddd8834579e31cf84f719be76be/pkg/cmd/roachtest/tests/kv.go#L353)). This causes nightly benchmark runs to use different storage configurations across runs, making performance comparisons unreliable. Similar regressions have occurred in the past (e.g., changing default machine types).
## Proposal
Introduce a "benchmark contract" mechanism that enforces deterministic hardware configurations for performance-tracked benchmarks:
1. **Declarative benchmark spec**: Define a struct or configuration that pins the full cluster spec for each benchmark (machine type, CPU, disk type, filesystem, number of disks, RAID config, cloud provider).
2. **CI lint/test enforcement**: Add a lint check or test that asserts benchmark-tagged tests (`Benchmark: true` in `TestSpec`) never use randomized spec options (e.g., `RandomizeVolumeType`, `RandomlyUseXfs`). This catches the class of bugs where "harmless" randomization or default changes silently break benchmark stability.
3. **Spec recording in results**: Record the full cluster spec alongside benchmark results so that configuration drift can be detected retroactively and discontinuities in performance timelines can be identified.
## Immediate fix
Remove `RandomizeVolumeType()` and `RandomlyUseXfs()` from nightly perf-tracked KV benchmarks. Randomization should be confined to weekly or dedicated correctness/robustness test variants.
## References
- Regression introduced at: [kv.go#L353](https://github.com/cockroachdb/cockroach/blob/90ba43f495776ddd8834579e31cf84f719be76be/pkg/cmd/roachtest/tests/kv.go#L353)
- `TestSpec.Benchmark` marker already exists in the registry and can be leveraged for enforcement.
Epic: none
Jira issue: CRDB-62985
Contributor guide
Assessment
This issue has not been assessed yet.