When Kafka sink cannot handle a large single-row txn (>100MiB), it is not possible to find out which txn-startTS is the cause from logs or cli

What did you do?

1. Prepare TiKV config with raft-entry-max-size increased to 64MiB to allow extra long fields.

# allow-long-fields.toml
[raftstore]
raft-entry-max-size = '64MiB'

2. Start a playground

tiup playground v8.5.3 --db 1 --kv 1 --pd 1 --tiflash 0 --ticdc 1 \
    --kv.config ./allow-long-fields.toml \
    --ticdc.config ./ticdc-newarch.toml \
    --ticdc.binpath ~/.tiup/components/cdc/v8.5.4-release.2/cdc

3. Create Kafka topic with larger max.message.bytes

./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic topic --config max.message.bytes=120000000

3. Create a changefeed to Kafka.

tiup cdc:v8.5.4-release.2 cli changefeed create -c test7 --sink-uri 'kafka://10.0.0.1:9092/topic?protocol=canal-json&max-message-bytes=120000000'

4. Create the table hosting large data.

use test;
create table j (
    pk bigint primary key,
    c0 json not null,
    c1 json not null,
    c2 json not null,
    c3 json not null,
    c4 json not null,
    c5 json not null,
    c6 json not null,
    c7 json not null,
    c8 json not null,
    c9 json not null
);

5. Insert the long row.

set tidb_txn_entry_size_limit = 64000000;
set @h = concat('[', repeat('"&",', 888888), '"&"]');
insert into j values (1, @h, @h, @h, @h, @h, @h, @h, @h, @h, @h);

6. Check the startTS and commitTS for reference.

tiup ctl:v8.5.3 tidb mvcc key -d test -t j -i 1 | cut -c 1-100

{
    "key": "7480000000000000705F728000000000000001",
    "region_id": 16,
    "value": {
        "info": {
            "writes": [
                {
                    "start_ts": 462008581595070469,
                    "commit_ts": 462008581791678469
                }
            ],
            "values": [
                {
                    "start_ts": 462008581595070469,
                    "value": "gAEKAAAAAgAAAAMAAAAEAAAABQAAAAYAAAAHAAAACAAAAAkAAAAKAAAACwAAAJjxXgAw47
                }
            ]
        }
    }
}

What did you expect to see?

Changefeed entered "warning" state ("[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic topic: kafka: error encoding packet: invalid request size (106667184)")

Yes sending this message out should not be possible because of sarama's hard-coded 100MiB limit, but the log should show me which row or txn (the startTS) is causing the error, so we can at least locate the row or skip it via ignore-txn-start-ts.

What did you see instead?

The log contains no mention of the startTS (462008581595070469). The only relevant timestamp is the checkpointTS (462008581791678468), which is the commitTS − 1. But we can't use the commitTS to find startTS (plus the commitTS is not unique), so it is not useful at all. We don't even know which table is causing the error.

[2025/11/06 17:54:35.646 +08:00] [ERROR] [dispatcher_manager.go:514] ["Event Dispatcher Manager Meets Error"] [changefeedID=default/test7] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic topic: kafka: error encoding packet: invalid request size (106667184)"]
[2025/11/06 17:54:35.652 +08:00] [ERROR] [maintainer.go:717] ["dispatcher report an error"] [changefeed=default/test7] [sourceNode=5928eff1-0acc-4a3d-8eb4-58cfebfd0d76] [error="[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic topic: kafka: error encoding packet: invalid request size (106667184)"]
[2025/11/06 17:54:35.717 +08:00] [ERROR] [backoff.go:157] ["changefeed maintainer report an error"] [changefeed=default/test7] [checkpointTs=462008581791678468] [node=127.0.0.1:8300] [code=CDC:ErrChangefeedRetryable] [state=warning] [error="time:\"2025-11-06 17:54:35.652262 +0800 HKT m=+1055.245365834\" node:\"127.0.0.1:8300\" code:\"CDC:ErrChangefeedRetryable\" message:\"[CDC:ErrKafkaAsyncSendMessage]kafka async send message failed: kafka: Failed to produce message to topic topic: kafka: error encoding packet: invalid request size (106667184)\" "]

The case of < 100 MiB, but exceeding max.message.bytes is actually similar, but at least we know which table is causing the error.

[2025/11/06 17:43:08.907 +08:00] [ERROR] [canal_json_encoder.go:467] ["Single message is too large for canal-json"] [maxMessageBytes=1048460] [length=49200587] [table=test.j]
...
[2025/11/06 17:43:08.908 +08:00] [ERROR] [dispatcher_manager.go:514] ["Event Dispatcher Manager Meets Error"] [changefeedID=default/test6] [error="[CDC:ErrMessageTooLarge]message is too large"]
[2025/11/06 17:43:08.908 +08:00] [ERROR] [maintainer.go:717] ["dispatcher report an error"] [changefeed=default/test6] [sourceNode=5928eff1-0acc-4a3d-8eb4-58cfebfd0d76] [error="[CDC:ErrMessageTooLarge]message is too large"]
[2025/11/06 17:43:08.968 +08:00] [ERROR] [backoff.go:157] ["changefeed maintainer report an error"] [changefeed=default/test6] [checkpointTs=462008370018648069] [node=127.0.0.1:8300] [code=CDC:ErrChangefeedRetryable] [state=warning] [error="time:\"2025-11-06 17:43:08.908237 +0800 HKT m=+368.450227043\" node:\"127.0.0.1:8300\" code:\"CDC:ErrChangefeedRetryable\" message:\"[CDC:ErrMessageTooLarge]message is too large\" "]

Versions of the cluster

v8.5.4-release.2

Start by tracing the Kafka error from dispatcher_manager.go and maintainer.go, then compare it with the table context logged in canal_json_encoder.go. Reproduce the large single-row transaction scenario and make the failure identify the affected table and txn startTS, including when Sarama rejects the oversized request.