debezium / debezium/dbz

Mysql Connector with ProxySQL [DBZ-2667]

Open
#375 0 comments 0 reactions 0 assignees View on GitHub
migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-2667](https://issues.redhat.com/browse/DBZ-2667)

Hello team,

 

I am trying to setup a Mysql Connector that connects with Mysql cluster with 2 replica nodes (I want the connector to read data from replicas only) though ProxySQL to achieve high availability -> but it did not work.

I also tried with HAProxy and it worked.

Is that possible to support the connector to connect to ProxySQL (as it's quite popular proxy for Mysql) or we have to go with HAProxy?

 

Here is the logs when the connector connects though ProxySQL
{code:java}
2020-10-16 07:55:57,073 INFO   MySQL|my_server_test|snapshot  'performance_schema.variables_info' is filtered out of capturing   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:57,073 INFO   MySQL|my_server_test|snapshot  'performance_schema.variables_info' is filtered out of capturing   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:57,077 INFO   MySQL|my_server_test|snapshot  including 'sys.sys_config' among known tables   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:57,078 INFO   MySQL|my_server_test|snapshot  'sys.sys_config' is filtered out of capturing   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:57,081 INFO   MySQL|my_server_test|snapshot  snapshot continuing with database(s): [my_db_test]   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:57,081 INFO   MySQL|my_server_test|snapshot  Step 6: generating DROP and CREATE statements to reflect current database schemas:   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:58,754 INFO   MySQL|my_server_test|snapshot  Step 7: releasing global read lock to enable MySQL writes   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:58,756 INFO   MySQL|my_server_test|snapshot  Step 7: blocked writes to MySQL for a total of 00:00:01.736   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:58,758 INFO   MySQL|my_server_test|snapshot  Step 8: scanning contents of 0 tables while still in transaction   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:58,759 INFO   MySQL|my_server_test|snapshot  Step 8: scanned 0 rows in 0 tables in 00:00:00.0   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:58,760 INFO   MySQL|my_server_test|snapshot  Step 9: committing transaction   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:58,762 INFO   MySQL|my_server_test|snapshot  Completed snapshot in 00:00:01.816   [io.debezium.connector.mysql.SnapshotReader]2020-10-16 07:55:58,925 INFO   MySQL|my_server_test|task  Transitioning from the snapshot reader to the binlog reader   [io.debezium.connector.mysql.ChainedReader]2020-10-16 07:55:58,926 WARN   MySQL|my_server_test|task  After applying blacklist/whitelist filters there are no tables to monitor, please check your configuration   [io.debezium.relational.RelationalDatabaseSchema]2020-10-16 07:55:58,943 INFO   MySQL|my_server_test|task  Creating thread debezium-mysqlconnector-my_server_test-binlog-client   [io.debezium.util.Threads]2020-10-16 07:55:58,946 INFO   MySQL|my_server_test|task  Creating thread debezium-mysqlconnector-my_server_test-binlog-client   [io.debezium.util.Threads]Oct 16, 2020 7:55:59 AM com.github.shyiko.mysql.binlog.BinaryLogClient connectINFO: Connected to 172.17.0.11:6033 at mysql-bin.000003/3815 (sid:2, cid:207)2020-10-16 07:55:59,089 INFO   MySQL|my_server_test|binlog  Connected to MySQL binlog at 172.17.0.11:6033, starting at binlog file 'mysql-bin.000003', pos=3815, skipping 0 events plus 0 rows   [io.debezium.connector.mysql.BinlogReader]2020-10-16 07:55:59,089 INFO   MySQL|my_server_test|task  Waiting for keepalive thread to start   [io.debezium.connector.mysql.BinlogReader]2020-10-16 07:55:59,091 INFO   MySQL|my_server_test|binlog  Creating thread debezium-mysqlconnector-my_server_test-binlog-client   [io.debezium.util.Threads]2020-10-16 07:55:59,129 INFO   MySQL|my_server_test|binlog  Stopped reading binlog after 0 events, no new offset was recorded   [io.debezium.connector.mysql.BinlogReader]2020-10-16 07:55:59,191 INFO   MySQL|my_server_test|task  Keepalive thread is running   [io.debezium.connector.mysql.BinlogReader]2020-10-16 07:56:56,381 INFO   ||  WorkerSourceTask{id=my-connector-mysql-test-ncic-0} Committing offsets   [org.apache.kafka.connect.runtime.WorkerSourceTask]2020-10-16 07:56:56,381 INFO   ||  WorkerSourceTask{id=my-connector-mysql-test-ncic-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]Oct 16, 2020 7:56:59 AM com.github.shyiko.mysql.binlog.BinaryLogClient$5 runINFO: Trying to restore lost connection to 172.17.0.11:60332020-10-16 07:56:59,093 INFO   MySQL|my_server_test|binlog  Creating thread debezium-mysqlconnector-my_server_test-binlog-client   [io.debezium.util.Threads]2020-10-16 07:56:59,229 INFO   MySQL|my_server_test|binlog  Connected to MySQL binlog at 172.17.0.11:6033, starting at binlog file 'mysql-bin.000003', pos=3815, skipping 0 events plus 0 rows   [io.debezium.connector.mysql.BinlogReader]Oct 16, 2020 7:56:59 AM com.github.shyiko.mysql.binlog.BinaryLogClient connectINFO: Connected to 172.17.0.11:6033 at mysql-bin.000003/3815 (sid:2, cid:208)2020-10-16 07:56:59,269 INFO   MySQL|my_server_test|binlog  Stopped reading binlog after 0 events, no new offset was recorded   [io.debezium.connector.mysql.BinlogReader]2020-10-16 07:57:56,382 INFO   ||  WorkerSourceTask{id=my-connector-mysql-test-ncic-0} Committing offsets   [org.apache.kafka.connect.runtime.WorkerSourceTask]2020-10-16 07:57:56,382 INFO   ||  WorkerSourceTask{id=my-connector-mysql-test-ncic-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]Oct 16, 2020 7:57:59 AM com.github.shyiko.mysql.binlog.BinaryLogClient$5 runINFO: Trying to restore lost connection to 172.17.0.11:60332020-10-16 07:57:59,231 INFO   MySQL|my_server_test|binlog  Creating thread debezium-mysqlconnector-my_server_test-binlog-client   [io.debezium.util.Threads]Oct 16, 2020 7:57:59 AM com.github.shyiko.mysql.binlog.BinaryLogClient connectINFO: Connected to 172.17.0.11:6033 at mysql-bin.000003/3815 (sid:2, cid:209)2020-10-16 07:57:59,365 INFO   MySQL|my_server_test|binlog  Connected to MySQL binlog at 172.17.0.11:6033, starting at binlog file 'mysql-bin.000003', pos=3815, skipping 0 events plus 0 rows   [io.debezium.connector.mysql.BinlogReader]2020-10-16 07:57:59,405 INFO   MySQL|my_server_test|binlog  Stopped reading binlog after 0 events, no new offset was recorded   [io.debezium.connector.mysql.BinlogReader]
{code}
 

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported setup with a MySQL cluster, two replica nodes, and ProxySQL, then compare it with the working HAProxy setup. Start with the logged SnapshotReader, BinlogReader, and BinaryLogClient behavior; done means determining whether ProxySQL can be supported and documenting the required behavior or limitations.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.