redpanda-data / redpanda-data/observability
Internal rpc latency differences
Nobody has claimed this yet.
- Dominant language
- Jsonnet
- Stars
- 52
- Forks
- 12
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 1
Description
internal_rpc latency is reported differently on the Default and Ops dashboard making it difficult to tell what the true p95 and p99 values are. E.g.
PromQL query for internal_rpc latency p95 on Ops Dashboard:
histogram_quantile(0.95, sum(rate(redpanda_rpc_request_latency_seconds_bucket{instance=~"[[node]]",exported_instance=~"[[exported_node]]",shard=~"[[node_shard]]",redpanda_cloud_data_cluster_name=~"[[data_cluster]]"}[5m])) by (le, [[aggr_criteria]]))
PromQL query for internal_rpc latency p95 on Default Dashboard:
histogram_quantile(0.95, sum(rate(redpanda_rpc_request_latency_seconds_bucket{instance=~"$node",redpanda_server="internal"}[$__rate_interval])) by (le, $aggr_criteria))
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare the internal_rpc p95 PromQL queries shown for the Ops and Default dashboards, including their label filters, aggregation criteria, and rate windows. Determine the intended common calculation for p95 and p99, then verify that both dashboards report consistent values.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100