cockroachdb / cockroachdb/cockroach
changefeedccl: corrupted resolved message
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
While doing testing for parity between the v1 and v2 kafka sinks, I encountered a case where 2 (*identical*) resolved messages were delivered to kafka corrupted. Here are the two messages:
```
$ cat -e ~/tmp/v1d.data | head
^@^@^@^@^B^B<1722630839671573555.0000000000$
^@^@^@^@^B^B<1722630839671573555.0000000000$
(..and then normal messages follow..)
{"resolved":"1722630948615128741.0000000000"}$
{"resolved":"1722630950782560018.0000000000"}$
```
(the above uses `cat -e` to visualize escape codes, as the corruption produces non-readable characters. this file is attached.)
`v1d.data` is the result of running `kcat -b milesfrankel-kafkatesting-0004.roachprod.crdb.io:9092 -t v1_district -Ce` which I believe to be reliable, and the results of which are reproducible.
This only happened for one topic, and these are the *first messages in that topic*.
I'll update this ticket if I am able to reproduce it, but I'll attach all the info I can.
Setup:
- master branch
- 3 cockroach nodes with a 4th machine running the tpcc workload on the cluster.
- 3 kafka nodes - topics set to replication = 3, num partitions = 3
- changefeed statement: `CREATE CHANGEFEED FOR TABLE warehouse, TABLE district, TABLE customer, TABLE history, TABLE "order", TABLE new_order, TABLE item, TABLE stock, TABLE order_line INTO 'kafka://milesfrankel-kafkatesting-0004.roachprod.crdb.io:9092?topic_prefix=v1_' WITH OPTIONS (diff, format = 'json', initial_scan = 'no', kafka_sink_config = '{"RequiredAcks": "ALL"}', metrics_label = 'v1', resolved, updated)`
- there was another changefeed that was running concurrently with similar settings but with the v2 sink. this is the mechanism i'm using for testing it.
[v1d.data.gz](https://github.com/user-attachments/files/16499036/v1d.data.gz)
Cockroach logs: https://drive.google.com/file/d/19hrPn6aMKuKxPHSfrxIvcg2m2HVsoLUL/view?usp=sharing
Jira issue: CRDB-40934
Contributor guide
Assessment
This issue has not been assessed yet.