cockroachdb / cockroachdb/cockroach

changefeedccl: kafka changefeeds may send resolved messages to only some partitions

Open
#122,666 2 comments 0 reactions 0 assignees View on GitHub
A-cdc branch-release-23.2 branch-release-24.1 C-bug regression T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

After https://github.com/cockroachdb/cockroach/commit/361c142f6a1dacdf91a8efeac181bd6934593efc merged, Kafka changefeeds could sometimes send resolved messages to only some instead of all partitions for a topic.

The reason for this is that we (arguably incorrectly) use the `Partition` field on `sarama.ProducerMessage` to assign the partition, when this field is supposed to be written to (and not read) by sarama:
https://github.com/cockroachdb/cockroach/blob/b4c8023d2ef7c4e4916359d0e692f3e60542fb5d/pkg/ccl/changefeedccl/sink_kafka.go#L416-L426

Before the commit above, we also supplied a custom partitioner to sarama that would read the `Partition` field, so this was fine:
https://github.com/cockroachdb/cockroach/blob/b4c8023d2ef7c4e4916359d0e692f3e60542fb5d/pkg/ccl/changefeedccl/sink_kafka.go#L744-L751

But after the commit, the custom partitioner we supplied no longer had this wrapper struct and so we reverted to the default sarama behavior when a `nil` key was provided, namely randomly picking a partition. This means that even though we're emitting O(partitions) number of resolved events, it's not guaranteed that each partition will receive one, although if the random partitioner is relatively uniform, the expected value would be one resolved event per partition.

Jira issue: CRDB-38017

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.