cockroachdb / cockroachdb/cockroach

roachtest: add rolling restart performance tests

Open
#103,149 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

We have seen rolling restarts can lead to negative performance impacts on workloads https://github.com/cockroachdb/cockroach/issues/101321.

There are no roachtests which run rolling restarts of a cockroach cluster and report the workload performance throughout.

**Describe the solution you'd like**
Create a set of roachtests which run a diverse set of workloads such as:

- write heavy `kv0/32kb` `index creation`
- comprehensive `TPCC`
- balanced `indexes`
- read heavy `kv95/scan`

Then report the workload performance as the rolling restart runs.

- P99, P99.9 latency
- throughput

It is important that the workload is sized to appropriately to not overload a cluster of size N-1 normally i.e. fix rate to run at 65-70% of max possible throughput on an N-1 sized cluster.

Optionally the test could mix assertions (while still exporting a roachperf stats.json) in, failing the test after completing if the throughput or latency degrade beyond a set limit.

These tests can either piggyback off of existing roachtest structures, such as the TPCC/KV tests or be created in a new file+structure.

**Additional context**

Existing rolling restart tests:

Running the movr workload
https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go#L467

Upgrade utility which rolling restarts a subset of nodes, called by schema changer tests and change-replicas
https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/version.go#L138-L138

Rolling drain, no workload
https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/quit.go#L45-L45

TPCC with upgrade

https://github.com/cockroachdb/cockroach/blob/f04439ccda905bf003f77dd341b1bd123075a332/pkg/cmd/roachtest/tests/quit.go#L45-L45

Related: https://github.com/cockroachdb/cockroach/issues/101315

Jira issue: CRDB-27852

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing roachtests in pkg/cmd/roachtest/tests/loss_of_quorum_recovery.go, version.go, and quit.go, along with the referenced TPCC and KV test structures. Define rolling-restart tests for the listed workloads that report P99/P99.9 latency and throughput, size load for an N-1 cluster, and preserve roachperf stats.json; optional degradation assertions should run after completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.