CDC: ErrMessageTooLarge does not take compression into account
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 464
- Forks
- 313
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 14
Description
What did you do?
- Prepare a Kafka server at 10.0.0.9:9092
- Create 1 topic:
- topic_small with max.message.bytes = 100000 (100KB)
- Create a changefeed with
- sink-uri =
kafka://10.0.0.9:9092/topic_small?protocol=canal-json&compression=gzip
- sink-uri =
- Execute the following upstream:
create table test.big (pk bigint primary key, content longtext not null); insert into test.big values (2, repeat('y', 100000));
What did you expect to see?
Because of compression the actual message being sent will be well below 100 KB, so Kafka should accept the message, and so should TiCDC.
What did you see instead?
TiCDC actually blocks the message saying "Single message is too large for canal-json", citing length being well above 100 KB.
Versions of the cluster
TiCDC version (execute cdc version):
v7.5.6
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported Kafka changefeed scenario with canal-json and gzip, then trace the ErrMessageTooLarge check and the compression handling it compares against. Done means a 100 KB upstream row can be sent when its compressed Kafka message is within the topic limit, while genuinely oversized messages are still rejected; add or update a regression test if the existing test structure covers this path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100