cockroachdb / cockroachdb/cockroach

backup: standardize online-restore perf measurement across roachtests

Open
#172,498 1 comment 0 reactions 0 assignees View on GitHub
A-disaster-recovery C-enhancement T-disaster-recovery
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Problem

Online-restore (OR) roachtests measure foreground performance inconsistently.
`online/perf-breakdown` samples per-op latency from the workload over time; the
other `workload=true` tests scrape a cluster-wide p95 from Prometheus and export
a single "time to within 1.25× of healthy p95". The two aren't comparable, and
neither normalizes across workload, data size, or cluster size.

## Proposal

Track three KPIs for every OR test, collected the same way, and fold the
`perf-breakdown` collector into all `workload=true` OR tests.

### KPIs

1. **Total excess-latency per GB-per-node (ms·s / (GB/node)).**
Define `excess(t) = latency(t) − latency(t_after_download)`: how much worse
latency is at time `t` than post-restore steady state. Integrate `excess` over
the restore (trapezoidal, 30s slices) → ms·s, then divide by per-node bytes
downloaded. One number for the total workload impact, comparable across
restore sizes and cluster sizes. This is the metric we currently choose as
our north star: the single number to move.

2. **Excess p50/p99 @ 1min/5min vs. post-restore baseline (ms).** Too much
latency is disqualifying even if brief — operators prefer a longer, gentler
restore over a faster, harsher one.

3. **Throughput (MB/s per node)** = bytes downloaded / #nodes / download
duration. Faster restores are better, all else equal.

### Standard collection

Extract the `perf-breakdown` collector and fold it into the shared restore driver
so every `workload=true` OR test emits these, without per-test work:

- A **compact, plot-friendly time series** artifact with a sample per 30s per op
capturing the elapsed time, download percent, op qps, op latency p50, p95 and
p99. The north-star is derived from these latency+qps pairs.
- **Snowflake KPI readings**: overall (qps-weighted avg across ops) and per-op
KPIs, plus per-op latencies at both 0/30/60/90/100% downloaded and 1min
post-download and at test-chosen elapsed times, e.g. 30s, 1min, 5min. Keep
emitting the existing point names (e.g. `agg_p50_download_30`) alongside any
new ones so current dashboards and history stay intact.
- **Opt-in statement bundles** at phase milestones (30/60/90/100% downloaded),
running off the sampling path so they never perturb the cadence, but enable
post-hoc digging into latency readings captured above.

### Sunset

Once the shared driver emits this, the existing 350 GB `workload=true` perf test
produces exactly what `perf-breakdown` was built for, so **retire
`online/perf-breakdown`**.

## Rollout

1. Land KPIs + compact log in `perf-breakdown` *(largely the current branch)*.
2. Extract the collector, KPI derivation, and bundle plugin into a library.
3. Wire into the shared restore driver; enable for `workload=true` tests.
4. Validate KPI comparability across ≥2 fixture sizes (per-node linearity;
whether 1min is enough for the baseline to settle).
5. Retire `online/perf-breakdown`.

## Open questions

- **p50 vs p99 for the excess-latency integral** — p50 is stable; p99 is the
real tail signal but noisy at 5s windows for low-QPS ops.
- **Per-node linearity** — first-order correct; validate empirically.
- **Baseline settling** — 1min may be short if compaction is still catching up.

Jira issue: CRDB-65742

Contributor guide

Open the contributing guide

Research direction

Start with the existing online/perf-breakdown collector and the shared restore driver described in the proposal. Trace how workload=true tests collect Prometheus readings and emit Snowflake points, then assess the rollout steps and open questions around baseline settling and p50 versus p99. Done means shared collection produces the specified time series and KPIs for workload=true tests, preserves existing point names, and allows online/perf-breakdown to be retired.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.