cockroachdb / cockroachdb/cockroach

changefeedccl: corrupted resolved message

Open
#128,305 1 comment 0 reactions 0 assignees View on GitHub
A-cdc branch-master C-bug T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

While doing testing for parity between the v1 and v2 kafka sinks, I encountered a case where 2 (*identical*) resolved messages were delivered to kafka corrupted. Here are the two messages:

```
$ cat -e ~/tmp/v1d.data | head
^@^@^@^@^B^B<1722630839671573555.0000000000$
^@^@^@^@^B^B<1722630839671573555.0000000000$
(..and then normal messages follow..)
{"resolved":"1722630948615128741.0000000000"}$
{"resolved":"1722630950782560018.0000000000"}$
```

(the above uses `cat -e` to visualize escape codes, as the corruption produces non-readable characters. this file is attached.)
`v1d.data` is the result of running `kcat -b milesfrankel-kafkatesting-0004.roachprod.crdb.io:9092 -t v1_district -Ce` which I believe to be reliable, and the results of which are reproducible.

This only happened for one topic, and these are the *first messages in that topic*.

I'll update this ticket if I am able to reproduce it, but I'll attach all the info I can.

Setup:
- master branch
- 3 cockroach nodes with a 4th machine running the tpcc workload on the cluster.
- 3 kafka nodes - topics set to replication = 3, num partitions = 3
- changefeed statement: `CREATE CHANGEFEED FOR TABLE warehouse, TABLE district, TABLE customer, TABLE history, TABLE "order", TABLE new_order, TABLE item, TABLE stock, TABLE order_line INTO 'kafka://milesfrankel-kafkatesting-0004.roachprod.crdb.io:9092?topic_prefix=v1_' WITH OPTIONS (diff, format = 'json', initial_scan = 'no', kafka_sink_config = '{"RequiredAcks": "ALL"}', metrics_label = 'v1', resolved, updated)`
- there was another changefeed that was running concurrently with similar settings but with the v2 sink. this is the mechanism i'm using for testing it.

[v1d.data.gz](https://github.com/user-attachments/files/16499036/v1d.data.gz)
Cockroach logs: https://drive.google.com/file/d/19hrPn6aMKuKxPHSfrxIvcg2m2HVsoLUL/view?usp=sharing

Jira issue: CRDB-40934

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.