cockroachdb / cockroachdb/cockroach
changefeedccl: improve observability into aggregator sink flushes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
We currently have a `changefeed.flushes` metric that counts the number of aggregator sink flushes and `changefeed.flush_hist_nanos` metric which builds a histogram of the duration of each flush, but we don't have any logs or metrics that tell us why a flush happened.
From a quick skim of the code, it seems like there are three main reasons we flush:
1. The aggregator is sending a progress update to the frontier
https://github.com/cockroachdb/cockroach/blob/2f8519c1ae5020614ee1616c829e1d5b3702f942/pkg/ccl/changefeedccl/changefeed_processors.go#L869
2. The blocking buffer is blocked
https://github.com/cockroachdb/cockroach/blob/2f8519c1ae5020614ee1616c829e1d5b3702f942/pkg/ccl/changefeedccl/changefeed_processors.go#L785-L786
https://github.com/cockroachdb/cockroach/blob/359593525b285317e8eb35b1b385c98352faaa3d/pkg/ccl/changefeedccl/kvevent/blocking_buffer.go#L135
3. The aggregator is preparing to send its shutdown checkpoint https://github.com/cockroachdb/cockroach/blob/2f8519c1ae5020614ee1616c829e1d5b3702f942/pkg/ccl/changefeedccl/changefeed_processors.go#L732
We should add more metrics (or logs) to help us distinguish these and any other reasons we flush. (Maybe something like `changefeed.flush.`.)
Jira issue: CRDB-41843
Epic CRDB-42868
Contributor guide
Assessment
This issue has not been assessed yet.