cockroachdb / cockroachdb/cockroach

changefeedccl: improve observability into aggregator sink flushes

Open
#130,019 1 comment 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement P-3 T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We currently have a `changefeed.flushes` metric that counts the number of aggregator sink flushes and `changefeed.flush_hist_nanos` metric which builds a histogram of the duration of each flush, but we don't have any logs or metrics that tell us why a flush happened.

From a quick skim of the code, it seems like there are three main reasons we flush:
1. The aggregator is sending a progress update to the frontier
https://github.com/cockroachdb/cockroach/blob/2f8519c1ae5020614ee1616c829e1d5b3702f942/pkg/ccl/changefeedccl/changefeed_processors.go#L869
2. The blocking buffer is blocked
https://github.com/cockroachdb/cockroach/blob/2f8519c1ae5020614ee1616c829e1d5b3702f942/pkg/ccl/changefeedccl/changefeed_processors.go#L785-L786
https://github.com/cockroachdb/cockroach/blob/359593525b285317e8eb35b1b385c98352faaa3d/pkg/ccl/changefeedccl/kvevent/blocking_buffer.go#L135
3. The aggregator is preparing to send its shutdown checkpoint https://github.com/cockroachdb/cockroach/blob/2f8519c1ae5020614ee1616c829e1d5b3702f942/pkg/ccl/changefeedccl/changefeed_processors.go#L732

We should add more metrics (or logs) to help us distinguish these and any other reasons we flush. (Maybe something like `changefeed.flush.`.)

Jira issue: CRDB-41843

Epic CRDB-42868

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.