cockroachdb / cockroachdb/cockroach
roachtest: add rolling restart performance tests
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
We have seen rolling restarts can lead to negative performance impacts on workloads https://github.com/cockroachdb/cockroach/issues/101321.
There are no roachtests which run rolling restarts of a cockroach cluster and report the workload performance throughout.
**Describe the solution you'd like**
Create a set of roachtests which run a diverse set of workloads such as:
- write heavy `kv0/32kb` `index creation`
- comprehensive `TPCC`
- balanced `indexes`
- read heavy `kv95/scan`
Then report the workload performance as the rolling restart runs.
- P99, P99.9 latency
- throughput
It is important that the workload is sized to appropriately to not overload a cluster of size N-1 normally i.e. fix rate to run at 65-70% of max possible throughput on an N-1 sized cluster.
Optionally the test could mix assertions (while still exporting a roachperf stats.json) in, failing the test after completing if the throughput or latency degrade beyond a set limit.
These tests can either piggyback off of existing roachtest structures, such as the TPCC/KV tests or be created in a new file+structure.
**Additional context**
Existing rolling restart tests:
Running the movr workload
https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go#L467
Upgrade utility which rolling restarts a subset of nodes, called by schema changer tests and change-replicas
https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/version.go#L138-L138
Rolling drain, no workload
https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/quit.go#L45-L45
TPCC with upgrade
https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/quit.go#L45-L45
Related: https://github.com/cockroachdb/cockroach/issues/101315
Jira issue: CRDB-27852
Contributor guide
Research direction
Start by reading the existing roachtests in pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go, version.go, and quit.go, along with the referenced TPCC and KV test structures. Define rolling-restart tests for the listed workloads that report P99/P99.9 latency and throughput, size load for an N-1 cluster, and preserve roachperf stats.json; optional degradation assertions should run after completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100