debezium / debezium/dbz

Replication error with MySql HA setup [DBZ-674]

Open
#136 0 comments 0 reactions 0 assignees View on GitHub
component/mysql-connector migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-674](https://issues.redhat.com/browse/DBZ-674)

Architecture overview:

Debezium is configured to access Mysql cluster with master-slave replication mode through HAProxy, GTID enabled.

At first run, everything works as expected.
When Mysql instance where Debezium and underlying mysql-binlog-connector-java are connected crashes, mysql binlog client will try to reconnect, connection will be directed to new instance but instance will have different position inside binary log and task will be terminated.

Mysql replication error code 1236 was occurred, here’s the log.

When kafka connect task is restarted, only GTID info is set to binary log client, without binlog filename and position info and everything works

{quote}
connect | 2018-03-26 13:44:02,384 INFO MySQL|dbserver1|binlog Stopped reading binlog after 12 events, last recorded offset: {ts_sec=1522071785, file=binlogfilename.000001, pos=12560, gtids=c197da1d-30f9-11e8-9aa3-0242ac1a0008:1-36, row=1, server_id=1327906210, event=2} [io.debezium.connector.mysql.BinlogReader]
connect | 2018-03-26 13:44:40,637 INFO || WorkerSourceTask{id=ride-connector-0} Committing offsets [org.apache.kafka.connect.runtime.WorkerSourceTask]
connect | 2018-03-26 13:44:40,637 INFO || WorkerSourceTask{id=ride-connector-0} flushing 0 outstanding messages for offset commit [org.apache.kafka.connect.runtime.WorkerSourceTask]
connect | Mar 26, 2018 1:44:41 PM com.github.shyiko.mysql.binlog.BinaryLogClient$5 run
connect | INFO: Trying to restore lost connection to haproxy:3306
connect | 2018-03-26 13:44:41,887 INFO MySQL|dbserver1|binlog Creating thread debezium-mysqlconnector-dbserver1-binlog-client [io.debezium.util.Threads]
connect | Mar 26, 2018 1:44:41 PM com.github.shyiko.mysql.binlog.BinaryLogClient connect
connect | INFO: Connected to haproxy:3306 at c197da1d-30f9-11e8-9aa3-0242ac1a0008:1-37 (sid:184054, cid:361)
connect | 2018-03-26 13:44:41,987 INFO MySQL|dbserver1|binlog Connected to MySQL binlog at haproxy:3306, starting at GTIDs c197da1d-30f9-11e8-9aa3-0242ac1a0008:1-37 and binlog file 'binlogfilename.000001', pos=12867, skipping 1 events plus 0 rows [io.debezium.connector.mysql.BinlogReader]
connect | 2018-03-26 13:44:42,033 ERROR MySQL|dbserver1|binlog Error during binlog processing. Last offset stored = {ts_sec=1522071785, file=binlogfilename.000001, pos=12560, gtids=c197da1d-30f9-11e8-9aa3-0242ac1a0008:1-36, row=1, server_id=1327906210, event=2}, binlog reader near position = binlogfilename.000001/123 [io.debezium.connector.mysql.BinlogReader]
connect | 2018-03-26 13:44:42,035 INFO || WorkerSourceTask{id=ride-connector-0} Committing offsets [org.apache.kafka.connect.runtime.WorkerSourceTask]
connect | 2018-03-26 13:44:42,035 INFO MySQL|dbserver1|binlog Stopped reading binlog after 12 events, last recorded offset: {ts_sec=1522071785, file=binlogfilename.000001, pos=12560, gtids=c197da1d-30f9-11e8-9aa3-0242ac1a0008:1-36, row=1, server_id=1327906210, event=2} [io.debezium.connector.mysql.BinlogReader]
{quote}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.