cockroachdb / cockroachdb/cockroach
roachtest: analyze unavailability distribution in `failover` tests
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The `failover` tests currently measure the pMax unavailability across multiple fail/recover cycles. This is trivial, since it's the pMax latency of the workload across both failures and recoveries.
It would be useful to know the unavailability distribution too, i.e. the minimum and average unavailability during failures. This is harder, because we can't just look at the minimum and average request latencies, since they'll be heavily skewed by normal successful requests executed outside of the failure interval.
What we're really interested in here is the pMax request latency of any request that experienced unavailability during a single failure, and then compute the unavailability distribution across the pMax of all individual failures for a run. We could optionally also gather a histogram of request latencies during a failure.
We could probably do this by specifying some minimum unavailability threshold (e.g. 1 second), and then collect the pMax latency seen during a continuous interval where requests saw latencies above the threshold. Then we'd compute population statistics for these pMaxes across the entire run.
It's unclear how we'd report this. Additional roachperf graphs would be ideal, but I don't believe roachperf supports that.
Jira issue: CRDB-28112
Contributor guide
Assessment
This issue has not been assessed yet.