cockroachdb / cockroachdb/cockroach
changefeedccl: kafka changefeeds may send resolved messages to only some partitions
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
After https://github.com/cockroachdb/cockroach/commit/361c142f6a1dacdf91a8efeac181bd6934593efc merged, Kafka changefeeds could sometimes send resolved messages to only some instead of all partitions for a topic.
The reason for this is that we (arguably incorrectly) use the `Partition` field on `sarama.ProducerMessage` to assign the partition, when this field is supposed to be written to (and not read) by sarama:
https://github.com/cockroachdb/cockroach/blob/b4c8023d2ef7c4e4916359d0e692f3e60542fb5d/pkg/ccl/changefeedccl/sink_kafka.go#L416-L426
Before the commit above, we also supplied a custom partitioner to sarama that would read the `Partition` field, so this was fine:
https://github.com/cockroachdb/cockroach/blob/b4c8023d2ef7c4e4916359d0e692f3e60542fb5d/pkg/ccl/changefeedccl/sink_kafka.go#L744-L751
But after the commit, the custom partitioner we supplied no longer had this wrapper struct and so we reverted to the default sarama behavior when a `nil` key was provided, namely randomly picking a partition. This means that even though we're emitting O(partitions) number of resolved events, it's not guaranteed that each partition will receive one, although if the random partitioner is relatively uniform, the expected value would be one resolved event per partition.
Jira issue: CRDB-38017
Contributor guide
Assessment
This issue has not been assessed yet.