cockroachdb / cockroachdb/cockroach

roachtest: analyze unavailability distribution in `failover` tests

Open
#103,654 0 comments 0 reactions 0 assignees View on GitHub
O-qa T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The `failover` tests currently measure the pMax unavailability across multiple fail/recover cycles. This is trivial, since it's the pMax latency of the workload across both failures and recoveries.

It would be useful to know the unavailability distribution too, i.e. the minimum and average unavailability during failures. This is harder, because we can't just look at the minimum and average request latencies, since they'll be heavily skewed by normal successful requests executed outside of the failure interval.

What we're really interested in here is the pMax request latency of any request that experienced unavailability during a single failure, and then compute the unavailability distribution across the pMax of all individual failures for a run. We could optionally also gather a histogram of request latencies during a failure.

We could probably do this by specifying some minimum unavailability threshold (e.g. 1 second), and then collect the pMax latency seen during a continuous interval where requests saw latencies above the threshold. Then we'd compute population statistics for these pMaxes across the entire run.

It's unclear how we'd report this. Additional roachperf graphs would be ideal, but I don't believe roachperf supports that.

Jira issue: CRDB-28112

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.