cockroachdb / cockroachdb/cockroach

kv: reconsider performance asserting roachtests

Open
#106,344 1 comment 0 reactions 0 assignees View on GitHub
A-kv C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
There are roachtests which assert that performance is no less than X, where X is usually a performance number the test gathers before changing the cluster (killing a node, restarting etc).

These tests are sensitive to flakes unrelated to the desired functionality being tested. These tests also reduce a higher fidelity performance number into a binary pass/fail -- losing information.

**Describe the solution you'd like**
Export performance of a run into `roachperf` and remove any failure conditions on performance.

These tests would be good candidates to start with:

- `kv/restart/nodes=12`
- `rebalance/by-load`
- `kv/quiescence/nodes=3`
- `kv/gracefuldraining/nodes=3`[^1] (see https://github.com/cockroachdb/cockroach/issues/87853)

[^1]: https://github.com/cockroachdb/cockroach/issues/106490

**Describe alternatives you've considered**

Performance tracking requires someone(or automated) to periodically check for regressions or improvements and flag them.

When a test fails, the artifacts and debug zip are collected. This won't happen automatically any longer, although we could change this.

We could instead keep the assertion and still collect metrics for viewing performance over time.

Jira issue: CRDB-29517

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.