cockroachdb / cockroachdb/cockroach
pua: benchmarks smooth over latency spikes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
The PUA benchmarks rely on DataDog for the metrics heavy lifting. In the raw timeseries data, one can clearly observe latency spikes during the intervals in which there are node restarts / kills and the loss of a zone / region. These latency spikes are unavoidable today, and should be shown in the results.
However, the latency spikes are not being shown in the benchmark results. This is likely due to the way in which the metrics backend (DataDog) is rolling up the latencies.
We should make sure that we are not averaging percentiles in the rollup. We should ensure that we preserve the blips (i.e. use `max` of the individual percentile calculations).
More conversation [here](https://cockroachlabs.slack.com/archives/C0988EKB8D7/p1767797827864949) (internal).
Jira issue: CRDB-58688
Contributor guide
Assessment
This issue has not been assessed yet.