cockroachdb / cockroachdb/cockroach
kv: reconsider performance asserting roachtests
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
There are roachtests which assert that performance is no less than X, where X is usually a performance number the test gathers before changing the cluster (killing a node, restarting etc).
These tests are sensitive to flakes unrelated to the desired functionality being tested. These tests also reduce a higher fidelity performance number into a binary pass/fail -- losing information.
**Describe the solution you'd like**
Export performance of a run into `roachperf` and remove any failure conditions on performance.
These tests would be good candidates to start with:
- `kv/restart/nodes=12`
- `rebalance/by-load`
- `kv/quiescence/nodes=3`
- `kv/gracefuldraining/nodes=3`[^1] (see https://github.com/cockroachdb/cockroach/issues/87853)
[^1]: https://github.com/cockroachdb/cockroach/issues/106490
**Describe alternatives you've considered**
Performance tracking requires someone(or automated) to periodically check for regressions or improvements and flag them.
When a test fails, the artifacts and debug zip are collected. This won't happen automatically any longer, although we could change this.
We could instead keep the assertion and still collect metrics for viewing performance over time.
Jira issue: CRDB-29517
Contributor guide
Assessment
This issue has not been assessed yet.