cockroachdb / cockroachdb/cockroach

Discrepancy in 30min interval metric visualisation

Open
#155,432 1 comment 0 reactions 0 assignees View on GitHub
C-bug T-supportability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

We have collected the tsdump with 30min metric interval for single node cluster. We compared the metrics dashboard widgets in both DB Console and Datadog uploaded tsdump. We observed that there is a discrepancy in the gauge and histogram values. We tried to capture the tsdump with text format and the values are matching with the Datadog uploaded values. We need to bring the parity between the metrics visualised in DB console and Datadog.

**To Reproduce**

- spin up the single node cluster with command `COCKROACH_DEBUG_TS_IMPORT_FILE=./cockroach/crl_30_min.raw ./cockroach/cockroach start-single-node --insecure --advertise-addr 127.0.0.1`
- upload the tsdump to Datadog with command `./cockroach debug tsdump --format datadog --dd-api-key $DD_API_KEY --cluster-label="crl-30-min" --dd-site=us5 crl_30_min.raw`
- Datadog dashboard will take couple of hours to load the data.
- Compare gauge and histogram metrics (DB console metric name vs Datadog metric name):
- `cr.node.sql.conns` vs `cockroachdb.sql.conns`
- `cr.node.exec.latency-p99` vs `cockroachdb.exec.latency_p99`

**Expected behavior**
The graphs should match for gauge and histogram metrics.

**Additional data / screenshots**

- [crl_30_min.raw.zip](https://github.com/user-attachments/files/22924840/crl_30_min.raw.zip)
- [crl_30_min.txt](https://github.com/user-attachments/files/22924848/crl_30_min.txt)
- `cr.node.sql.conns` vs `cockroachdb.sql.conns`
Image
Image

- `cr.node.exec.latency-p99` vs `cockroachdb.exec.latency_p99`
Image
Image

Jira issue: CRDB-55433

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.