Bypass MySQL missing or corrupted binlog --> use next one [DBZ-5892]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-5892](https://issues.redhat.com/browse/DBZ-5892)
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
We had a MySQL corruption and we see see which binlog the Connectors try to consume but due to error on partially data could be extracted with mysqlbinlog
We have see in the doc the following:
Sometimes while doing experiments (or when a connector was misconfigured at the start) it is necessary to remove the connector offsets to start with a clean state.
The first step is to find out the name of the topic that contains plugin-offsets. This is configured in {{offset.storage.topic}} option.
The next step is to find out the last offset for the given connector, key under which it is stored and identify the partition used to store the offset. An example would be:
{{$ kafkacat -b localhost -C -t my_connect_offsets -f 'Partition(%p) %k %s\n'Partition(11) ["inventory-connector",{"server":"dbserver1"}] {"ts_sec":1530088501,"file":"mysql-bin.000003","pos":817,"row":1,"server_id":223344,"event":2}
Partition(11) ["inventory-connector",{"server":"dbserver1"}] {"ts_sec":1530168941,"file":"mysql-bin.000004","pos":3261,"row":1,"server_id":223344,"event":2}}}
The key for {{inventory-connector}} is {{{}["inventory-connector",\{"server":"dbserver1"}]{}}}, the partition number is {{11}} and the last offset is {{{}{"ts_sec":1530168941,"file":"mysql-bin.000004","pos":3261,"row":1,"server_id":223344,"event":2}{}}}.
To delete connector offsets the connector should be stopped and the following command has to be issued:
{{$ echo '["inventory-connector",\{"server":"dbserver1"}]|' | \kafkacat -P -Z -b localhost -t my_connect_offsets -K \| -p 11}}
{{}}
{{so could it possible to execute this in our Kafka-debezium to tell the connector to skip this binlog ?}}
{{}}
{{What should be set the position of the next one ?}}
{{}}
We see this with the command kcat:
Partition(6) ["srt_prd_01",\{"server":"srt_prd_01"}] \{"transaction_id":null,"ts_sec":1669626494,"file":"mysql-bin.033874","pos":39784690,"row":1,"server_id":5,"event":2}
Partition(15) ["srt_prd",\{"server":"srt_prd"}] \{"transaction_id":null,"ts_sec":1667555091,"file":"mysql-bin.033594","pos":70099918,"row":1,"server_id":5,"event":2}
In the connector error log:
[2022-12-01 11:18:32,480] INFO Creating thread debezium-mysqlconnector-srt_prd_01-binlog-client (io.debezium.util.Threads)
[2022-12-01 11:18:32,480] INFO Waiting for keepalive thread to start (io.debezium.connector.mysql.MySqlStreamingChangeEventSource)
[2022-12-01 11:18:32,481] ERROR Producer failure (io.debezium.pipeline.ErrorHandler)
io.debezium.DebeziumException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'mysql-bin.033874' at 49111781, the last event read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719375, the last byte read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719394. Error code: 1236; SQLSTATE: HY000.
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1172)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1217)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:980)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.github.shyiko.mysql.binlog.network.ServerException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'mysql-bin.033874' at 49111781, the last event read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719375, the last byte read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719394.
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:944)
(max size is already set to 1GB) and with myslqbinlog tool we have the following when we go over position 49719313
mysqlbinlog --no-defaults mysql-bin.033874 > test.log
ERROR: Error in Log_event::read_log_event(): 'read error', data_len: 3708331197, event_type: -124
ERROR: Could not read entry at offset 49719375: Error in log format or read error.
Regards,
Contributor guide
Research direction
Start with MySqlStreamingChangeEventSource.java and the BinaryLogClient stack trace, then reproduce the failure using mysqlbinlog around the reported offset in mysql-bin.033874. Compare the connector's recorded offsets from kcat with the corrupted event and define the expected behavior for continuing with a later binlog and its starting position.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100