cockroachdb / cockroachdb/cockroach
changefeed: Add hard cap on batch size for non-kafka sinks
Open
A-cdc
C-enhancement
T-cdc
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Our recommended tuning for the batching sink is to set `Flush.MaxBytes` (kafka) and `Flush.Bytes` (webhook/pubsub) as high as the sink can handle, combined with a `Flush.Frequency` to cap latency. This recommendation would be cleaner if there were a **hard cap** on batch size for non-kafka sinks — `Flush.Bytes` currently does not behave as one.
Expose this either with a `Flush.MaxBytes` option or change the existing `Flush.Bytes` behavior if backwards compatibility allows.
Related: #165386
Jira issue: CRDB-64468
Contributor guide
Assessment
This issue has not been assessed yet.