cockroachdb / cockroachdb/cockroach

cdc: Add metric to track max age of in-flight sink flushes

Open
#164,450 1 comment 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**

When webhook sink requests timeout or hang, existing metrics (BatchHistNanos, CommitLatency) aren't recorded since they only trigger on successful flushes. This makes it difficult to diagnose whether a lagging changefeed is caused by sink issues versus upstream problems. We have encountered this issue when diagnosing issues in CC.

**Describe the solution you'd like**
Track the age of the oldest batch currently being flushed to the sink. This metric would continuously increase for batches that are stuck in the sink, even when they eventually timeout, making sink-related lag visible. We should be able to look at this metric rising to confirm that the changefeed lag is due to the sink.

**Describe alternatives you've considered**
Our best current proxy for this is cockroachdb.changefeed.buffer_entries.in/out and seeing if it drops to 0, but this is imprecise and sometimes hard in datadog.

**Additional context**
This came up with respect to the investigation discussed here: https://cockroachlabs.slack.com/archives/C01L5LYD401/p1772043698323869?thread_ts=1772039935.930829&cid=C01L5LYD401.

Jira issue: CRDB-60807

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.