Support non-full format mysql binary logs to identify table changes [DBZ-1343]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-1343](https://issues.redhat.com/browse/DBZ-1343)
The setting for my.cnf are:
log-bin=/var/log/mysql/mysql-bin
binlog_format=row
*binlog_row_image=full*
As per the [mysql documentation|https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_image] , if the engine type is `NDB`, mysql logs only `minimal` changes to log files. Even if the binlog_row_image is set to `full`.
This causes no issues, if the Insert or Update query affects all columns. But, if you have an insert or update to only partial columns, the `minimal` feature means that partial data is posted on the log.
Full insert:
{noformat}
### INSERT INTO `user_registrations`
### SET
### @1=824
### @2='+190554937263'
### @3='56DA9A2FD4D91320D01514ED497C719E5DC9323F'
### @4=1
### @5=NULL
### @6='2019-06-18 06:20:23'
### @7='2019-06-18 06:20:23'
### @8=NULL
{noformat}
Partial insert on the same (as in the bin log):
{noformat}
### INSERT INTO `user_registrations`
### SET
### @1=824
### @4=2
### @6='2019-06-18 06:20:26'
{noformat}
This results in casting exception in TableSchemaBuilder, since this is expecting a full row from the mysql connector.
Table Structure:
{noformat}
'id', 'int(10) unsigned', 'NO', 'PRI', NULL, 'auto_increment'
'msisdn', 'varchar(50)', 'NO', '', NULL, ''
'device_id', 'varchar(50)', 'NO', '', NULL, ''
'status', 'enum(\'waiting_for_validation\',\'approved\',\'rejected\',\'expired\')', 'NO', '', NULL, ''
'rejection_reason', 'enum(\'max_otp_validations_reached\',\'canceled_by_user\')', 'YES', '', NULL, ''
'updated_at', 'datetime', 'YES', '', 'CURRENT_TIMESTAMP', 'on update CURRENT_TIMESTAMP'
'created_at', 'datetime', 'NO', '', NULL, ''
'channel', 'varchar(50)', 'YES', '', NULL, '
{noformat}
* * Log trace looks as below:*
{noformat}
[2019-06-18 11:32:38,175] ERROR Failed to properly convert data value for 'wallet_db_v3.user_registrations.msisdn' of type VARCHAR for *row [824, 2, 2019-06-18T06:20:26]*: (io.debezium.relational.TableSchemaBuilder:253)
java.lang.IllegalArgumentException: Unexpected value for JDBC type 12 and column msisdn VARCHAR(50) CHARSET utf8 NOT NULL: class=class java.lang.Integer
at io.debezium.jdbc.JdbcValueConverters.handleUnknownData(JdbcValueConverters.java:1164)
at io.debezium.jdbc.JdbcValueConverters.convertValue(JdbcValueConverters.java:1207)
at io.debezium.connector.mysql.MySqlValueConverters.convertString(MySqlValueConverters.java:380)
at io.debezium.connector.mysql.MySqlValueConverters.lambda$converter$13(MySqlValueConverters.java:293)
at io.debezium.relational.TableSchemaBuilder.lambda$createValueGenerator$2(TableSchemaBuilder.java:249)
at io.debezium.relational.TableSchema.valueFromColumnData(TableSchema.java:145)
at io.debezium.connector.mysql.RecordMakers$1.insert(RecordMakers.java:244)
at io.debezium.connector.mysql.RecordMakers$RecordsForTable.create(RecordMakers.java:456)
at io.debezium.connector.mysql.BinlogReader.handleInsert(BinlogReader.java:833)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:501)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Contributor guide
Research direction
Start with the MySQL connector path named in the trace, especially BinlogReader, RecordMakers, TableSchemaBuilder, and MySqlValueConverters. Reproduce the shown partial INSERT or UPDATE binlog row and trace how its column values are converted. Done means the partial row no longer causes the reported conversion exception and the resulting table change is handled correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100