debezium / debezium/dbz

Support non-full format mysql binary logs to identify table changes [DBZ-1343]

Open
#229 0 comments 0 reactions 0 assignees View on GitHub
component/mysql-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
9
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-1343](https://issues.redhat.com/browse/DBZ-1343)

The setting for my.cnf are:

log-bin=/var/log/mysql/mysql-bin
binlog_format=row
*binlog_row_image=full*

As per the [mysql documentation|https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_image] , if the engine type is `NDB`, mysql logs only `minimal` changes to log files. Even if the binlog_row_image is set to `full`.

This causes no issues, if the Insert or Update query affects all columns. But, if you have an insert or update to only partial columns, the `minimal` feature means that partial data is posted on the log.

Full insert:
{noformat}
### INSERT INTO `user_registrations`
### SET
### @1=824
### @2='+190554937263'
### @3='56DA9A2FD4D91320D01514ED497C719E5DC9323F'
### @4=1
### @5=NULL
### @6='2019-06-18 06:20:23'
### @7='2019-06-18 06:20:23'
### @8=NULL
{noformat}
Partial insert on the same (as in the bin log):
{noformat}
### INSERT INTO `user_registrations`
### SET
### @1=824
### @4=2
### @6='2019-06-18 06:20:26'
{noformat}
This results in casting exception in TableSchemaBuilder, since this is expecting a full row from the mysql connector.

Table Structure:
{noformat}
'id', 'int(10) unsigned', 'NO', 'PRI', NULL, 'auto_increment'
'msisdn', 'varchar(50)', 'NO', '', NULL, ''
'device_id', 'varchar(50)', 'NO', '', NULL, ''
'status', 'enum(\'waiting_for_validation\',\'approved\',\'rejected\',\'expired\')', 'NO', '', NULL, ''
'rejection_reason', 'enum(\'max_otp_validations_reached\',\'canceled_by_user\')', 'YES', '', NULL, ''
'updated_at', 'datetime', 'YES', '', 'CURRENT_TIMESTAMP', 'on update CURRENT_TIMESTAMP'
'created_at', 'datetime', 'NO', '', NULL, ''
'channel', 'varchar(50)', 'YES', '', NULL, '
{noformat}

* * Log trace looks as below:*

{noformat}
[2019-06-18 11:32:38,175] ERROR Failed to properly convert data value for 'wallet_db_v3.user_registrations.msisdn' of type VARCHAR for *row [824, 2, 2019-06-18T06:20:26]*: (io.debezium.relational.TableSchemaBuilder:253)
java.lang.IllegalArgumentException: Unexpected value for JDBC type 12 and column msisdn VARCHAR(50) CHARSET utf8 NOT NULL: class=class java.lang.Integer
at io.debezium.jdbc.JdbcValueConverters.handleUnknownData(JdbcValueConverters.java:1164)
at io.debezium.jdbc.JdbcValueConverters.convertValue(JdbcValueConverters.java:1207)
at io.debezium.connector.mysql.MySqlValueConverters.convertString(MySqlValueConverters.java:380)
at io.debezium.connector.mysql.MySqlValueConverters.lambda$converter$13(MySqlValueConverters.java:293)
at io.debezium.relational.TableSchemaBuilder.lambda$createValueGenerator$2(TableSchemaBuilder.java:249)
at io.debezium.relational.TableSchema.valueFromColumnData(TableSchema.java:145)
at io.debezium.connector.mysql.RecordMakers$1.insert(RecordMakers.java:244)
at io.debezium.connector.mysql.RecordMakers$RecordsForTable.create(RecordMakers.java:456)
at io.debezium.connector.mysql.BinlogReader.handleInsert(BinlogReader.java:833)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:501)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)
{noformat}

Contributor guide

Open the contributing guide

Research direction

Start with the MySQL connector path named in the trace, especially BinlogReader, RecordMakers, TableSchemaBuilder, and MySqlValueConverters. Reproduce the shown partial INSERT or UPDATE binlog row and trace how its column values are converted. Done means the partial row no longer causes the reported conversion exception and the resulting table change is handled correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.