debezium / debezium/dbz

Bypass MySQL missing or corrupted binlog --> use next one [DBZ-5892]

Open
#730 0 comments 0 reactions 0 assignees View on GitHub
migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-5892](https://issues.redhat.com/browse/DBZ-5892)

In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

 

We had a MySQL corruption and we see see which binlog the Connectors try to consume but due to error on partially data could be extracted with mysqlbinlog 

 

We have see in the doc the following:
Sometimes while doing experiments (or when a connector was misconfigured at the start) it is necessary to remove the connector offsets to start with a clean state.
The first step is to find out the name of the topic that contains plugin-offsets. This is configured in {{offset.storage.topic}} option.
The next step is to find out the last offset for the given connector, key under which it is stored and identify the partition used to store the offset. An example would be:
 {{$ kafkacat -b localhost -C -t my_connect_offsets -f 'Partition(%p) %k %s\n'Partition(11) ["inventory-connector",{"server":"dbserver1"}] {"ts_sec":1530088501,"file":"mysql-bin.000003","pos":817,"row":1,"server_id":223344,"event":2}
Partition(11) ["inventory-connector",{"server":"dbserver1"}] {"ts_sec":1530168941,"file":"mysql-bin.000004","pos":3261,"row":1,"server_id":223344,"event":2}}}
The key for {{inventory-connector}} is {{{}["inventory-connector",\{"server":"dbserver1"}]{}}}, the partition number is {{11}} and the last offset is {{{}{"ts_sec":1530168941,"file":"mysql-bin.000004","pos":3261,"row":1,"server_id":223344,"event":2}{}}}.
To delete connector offsets the connector should be stopped and the following command has to be issued:
 {{$ echo '["inventory-connector",\{"server":"dbserver1"}]|' | \kafkacat -P -Z -b localhost -t my_connect_offsets -K \| -p 11}}
{{}}
{{so could it possible to execute this in our Kafka-debezium to tell the connector to skip this binlog ?}}
{{}}
{{What should be set the position of the next one ?}}
{{}}
We see this with the command kcat:
Partition(6) ["srt_prd_01",\{"server":"srt_prd_01"}] \{"transaction_id":null,"ts_sec":1669626494,"file":"mysql-bin.033874","pos":39784690,"row":1,"server_id":5,"event":2}
Partition(15) ["srt_prd",\{"server":"srt_prd"}] \{"transaction_id":null,"ts_sec":1667555091,"file":"mysql-bin.033594","pos":70099918,"row":1,"server_id":5,"event":2}
 
In the connector error log:
[2022-12-01 11:18:32,480] INFO Creating thread debezium-mysqlconnector-srt_prd_01-binlog-client (io.debezium.util.Threads)
[2022-12-01 11:18:32,480] INFO Waiting for keepalive thread to start (io.debezium.connector.mysql.MySqlStreamingChangeEventSource)
[2022-12-01 11:18:32,481] ERROR Producer failure (io.debezium.pipeline.ErrorHandler)
io.debezium.DebeziumException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'mysql-bin.033874' at 49111781, the last event read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719375, the last byte read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719394. Error code: 1236; SQLSTATE: HY000.
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1172)
        at io.debezium.connector.mysql.MySqlStreamingChangeEventSource$ReaderThreadLifecycleListener.onCommunicationFailure(MySqlStreamingChangeEventSource.java:1217)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:980)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
        at java.lang.Thread.run(Thread.java:750)
Caused by: com.github.shyiko.mysql.binlog.network.ServerException: log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'mysql-bin.033874' at 49111781, the last event read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719375, the last byte read from '/var/lib/mysql/logbin/mysql-bin.033874' at 49719394.
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:944)
 
(max size is already set to 1GB) and with myslqbinlog tool we have the following when we go over position 49719313 
mysqlbinlog  --no-defaults mysql-bin.033874  > test.log
ERROR: Error in Log_event::read_log_event(): 'read error', data_len: 3708331197, event_type: -124
ERROR: Could not read entry at offset 49719375: Error in log format or read error.

 
Regards,

Contributor guide

Open the contributing guide

Research direction

Start with MySqlStreamingChangeEventSource.java and the BinaryLogClient stack trace, then reproduce the failure using mysqlbinlog around the reported offset in mysql-bin.033874. Compare the connector's recorded offsets from kcat with the corrupted event and define the expected behavior for continuing with a later binlog and its starting position.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.