cockroachdb / cockroachdb/cockroach

pua: benchmarks smooth over latency spikes

Open
#161,068 0 comments 0 reactions 1 assignee Claimed by @BabuSrithar View on GitHub
branch-master C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

The PUA benchmarks rely on DataDog for the metrics heavy lifting. In the raw timeseries data, one can clearly observe latency spikes during the intervals in which there are node restarts / kills and the loss of a zone / region. These latency spikes are unavoidable today, and should be shown in the results.

However, the latency spikes are not being shown in the benchmark results. This is likely due to the way in which the metrics backend (DataDog) is rolling up the latencies.

We should make sure that we are not averaging percentiles in the rollup. We should ensure that we preserve the blips (i.e. use `max` of the individual percentile calculations).

More conversation [here](https://cockroachlabs.slack.com/archives/C0988EKB8D7/p1767797827864949) (internal).

Jira issue: CRDB-58688

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.