[Bug] Paimon mysql cdc When the mysql table structure changes, the task checkpoint cannot be submitted and is in the IN_PROGRESS state
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon-flink-1.17-0.8.jar
paimon-flink-action-0.8.0.jar
paimon-hive-connector-2.3-0.8.0.jar
### Compute Engine
flink version -> 1.17.2
hadoop -> 3.3.5
### Minimal reproduce step
1、create test table
CREATE TABLE `data_management`.`sql_template_info_paimon` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id',
`data_source_id` int(11) NOT NULL DEFAULT '1' COMMENT '数据源id, 1代表hive',
`table_name` varchar(100) NOT NULL COMMENT '当前操作的表名',
`chinese_table_name` varchar(1000) NOT NULL DEFAULT '' COMMENT '当前操作的中文表名',
`dimensionality` varchar(20) NOT NULL COMMENT '当前表的维度',
`table_in_tier` varchar(20) NOT NULL COMMENT '操作表所在层',
`is_partition_table` tinyint(1) NOT NULL COMMENT '是否为分区表',
`table_partition_type` varchar(100) NOT NULL COMMENT '表分区类型',
`table_importance_level` varchar(20) NOT NULL COMMENT '表重要等级',
`table_sensitivity_level` varchar(20) NOT NULL COMMENT '表敏感级别',
`partition_is_save_forever` tinyint(1) NOT NULL DEFAULT '0' COMMENT '分区是否永久保存',
`partition_keep_days` int(11) NOT NULL DEFAULT '7' COMMENT '分区保留天数,-1表示永久保存',
`update_frequency_level` varchar(20) NOT NULL COMMENT '更新频率',
`remark` longtext COMMENT '备注',
`old_table_name` varchar(200) DEFAULT NULL COMMENT '旧表名, 半角逗号隔开',
`define_label` varchar(100) DEFAULT NULL COMMENT '用户自定义标签,多个标签逗号隔开',
`heat` int(10) unsigned NOT NULL DEFAULT '0' COMMENT '热度',
`principal` varchar(60) NOT NULL COMMENT '当前表负责人',
`process_id` int(10) DEFAULT NULL COMMENT '工作流id',
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
`special_date_save_partition` varchar(255) DEFAULT '' COMMENT '节假日保存分区信息'
PRIMARY KEY (`id`),
UNIQUE KEY `uniq_n` (`table_name`),
KEY `idx_process_id` (`process_id`)
) ENGINE=InnoDB AUTO_INCREMENT=68318 DEFAULT CHARSET=utf8 COMMENT='模板对应的基本信息表'
2、Synchronize tasks through "mysql_sync_database"
sudo ./bin/flink run --allowNonRestoredState /home/q/module/flink/flink-1.17.2/lib/paimon-flink-action-0.8.0.jar mysql_sync_database --warehouse viewfs://qunarcluster/user/tujiadev/home/data/paimon/warehouse --type_mapping tinyint1-not-bool,to-nullable,char-to-string,bigint-unsigned-to-bigint --database tujia_ods --mode combined --merge_shards false --table_prefix ods_ --table_suffix _paimon --mysql_conf hostname=xx --mysql_conf username=xx --mysql_conf password=xx --mysql_conf database-name='data_management' --mysql_conf port=3310 --mysql_conf server-id=6289-6298 --mysql_conf scan.incremental.snapshot.chunk.size=80960 --table_conf bucket=10 --table_conf sink.parallelism=1 --table_conf file.format=parquet --table_conf file.compression=snappy --table_conf scan.infer-parallelism=false --table_conf scan.parallelism=100 --including_tables 'mail_tasks' --catalog_conf metastore=hive --catalog_conf uri='thrift://l-hiveserver4.data.cn2:9083,thrift://l-hiveserver5.data.cn2:9083,thrift://l-hiveserver6.data.cn2:9083,thrift://l-hiveserver7.data.cn2:9083,thrift://l-hiveserver8.data.cn2:9083'
3、The sql submitted by mysql is as follows:
①、 "CREATE TABLE `data_management`.`_sql_template_info_paimon_new` (换行 `id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id',换行 `data_source_id` int(11) NOT NULL DEFAULT '1' COMMENT '数据源id, 1代表hive',换行 `table_name` varchar(100) NOT NULL COMMENT '当前操作的表名',换行 `chinese_table_name` varchar(1000) NOT NULL DEFAULT '' COMMENT '当前操作的中文表名',换行 `dimensionality` varchar(20) NOT NULL COMMENT '当前表的维度',换行 `table_in_tier` varchar(20) NOT NULL COMMENT '操作表所在层',换行 `is_partition_table` tinyint(1) NOT NULL COMMENT '是否为分区表',换行 `table_partition_type` varchar(100) NOT NULL COMMENT '表分区类型',换行 `table_importance_level` varchar(20) NOT NULL COMMENT '表重要等级',换行 `table_sensitivity_level` varchar(20) NOT NULL COMMENT '表敏感级别',换行 `partition_is_save_forever` tinyint(1) NOT NULL DEFAULT '0' COMMENT '分区是否永久保存',换行 `partition_keep_days` int(11) NOT NULL DEFAULT '7' COMMENT '分区保留天数,-1表示永久保存',换行 `update_frequency_level` varchar(20) NOT NULL COMMENT '更新频率',换行 `remark` longtext COMMENT '备注',换行 `old_table_name` varchar(200) DEFAULT NULL COMMENT '旧表名, 半角逗号隔开',换行 `define_label` varchar(100) DEFAULT NULL COMMENT '用户自定义标签,多个标签逗号隔开',换行 `heat` int(10) unsigned NOT NULL DEFAULT '0' COMMENT '热度',换行 `principal` varchar(60) NOT NULL COMMENT '当前表负责人',换行 `process_id` int(10) DEFAULT NULL COMMENT '工作流id',换行 `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',换行 `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',换行 `special_date_save_partition` varchar(255) DEFAULT '' COMMENT '节假日保存分区信息',换行 PRIMARY KEY (`id`),换行 UNIQUE KEY `uniq_n` (`table_name`),换行 KEY `idx_process_id` (`process_id`)换行) ENGINE=InnoDB AUTO_INCREMENT=68315 DEFAULT CHARSET=utf8 COMMENT='模板对应的基本信息表'",
②、 "ALTER TABLE `data_management`.`_sql_template_info_paimon_new` ADD COLUMN paimon_field VARCHAR(20) DEFAULT '' NOT NULL COMMENT '辨别'"
③、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_del` AFTER DELETE ON `data_management`.`sql_template_info_paimon` FOR EACH ROW DELETE IGNORE FROM `data_management`.`_sql_template_info_paimon_new` WHERE `data_management`.`_sql_template_info_paimon_new`.`id` <=> OLD.`id`"
④、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_upd` AFTER UPDATE ON `data_management`.`sql_template_info_paimon` FOR EACH ROW REPLACE INTO `data_management`.`_sql_template_info_paimon_new` (`id`, `data_source_id`, `table_name`, `chinese_table_name`, `dimensionality`, `table_in_tier`, `is_partition_table`, `table_partition_type`, `table_importance_level`, `table_sensitivity_level`, `partition_is_save_forever`, `partition_keep_days`, `update_frequency_level`, `remark`, `old_table_name`, `define_label`, `heat`, `principal`, `process_id`, `create_time`, `update_time`, `special_date_save_partition`) VALUES (NEW.`id`, NEW.`data_source_id`, NEW.`table_name`, NEW.`chinese_table_name`, NEW.`dimensionality`, NEW.`table_in_tier`, NEW.`is_partition_table`, NEW.`table_partition_type`, NEW.`table_importance_level`, NEW.`table_sensitivity_level`, NEW.`partition_is_save_forever`, NEW.`partition_keep_days`, NEW.`update_frequency_level`, NEW.`remark`, NEW.`old_table_name`, NEW.`define_label`, NEW.`heat`, NEW.`principal`, NEW.`process_id`, NEW.`create_time`, NEW.`update_time`, NEW.`special_date_save_partition`)"
⑤、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_ins` AFTER INSERT ON `data_management`.`sql_template_info_paimon` FOR EACH ROW REPLACE INTO `data_management`.`_sql_template_info_paimon_new` (`id`, `data_source_id`, `table_name`, `chinese_table_name`, `dimensionality`, `table_in_tier`, `is_partition_table`, `table_partition_type`, `table_importance_level`, `table_sensitivity_level`, `partition_is_save_forever`, `partition_keep_days`, `update_frequency_level`, `remark`, `old_table_name`, `define_label`, `heat`, `principal`, `process_id`, `create_time`, `update_time`, `special_date_save_partition`) VALUES (NEW.`id`, NEW.`data_source_id`, NEW.`table_name`, NEW.`chinese_table_name`, NEW.`dimensionality`, NEW.`table_in_tier`, NEW.`is_partition_table`, NEW.`table_partition_type`, NEW.`table_importance_level`, NEW.`table_sensitivity_level`, NEW.`partition_is_save_forever`, NEW.`partition_keep_days`, NEW.`update_frequency_level`, NEW.`remark`, NEW.`old_table_name`, NEW.`define_label`, NEW.`heat`, NEW.`principal`, NEW.`process_id`, NEW.`create_time`, NEW.`update_time`, NEW.`special_date_save_partition`)"
⑥、 "ANALYZE TABLE `data_management`.`_sql_template_info_paimon_new` /* pt-online-schema-change */"
"RENAME TABLE `data_management`.`sql_template_info_paimon` TO `data_management`.`_sql_template_info_paimon_old`"
⑦、"DROP TABLE IF EXISTS `data_management`.`_sql_template_info_paimon_old`"
⑧、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_del`"
⑨、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_upd`"
⑩、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_ins`"
4、Insert a random piece of data into this table
5、After the binlog consumes the current insert data, you can see that checkpoint is in the IN_PROGRESS state indefinitely
### What doesn't meet your expectations?
After the synchronized mysql table structure changes, the paimon table cannot capture the change, and most importantly, the entire task is stuck
### Anything else?
The taskmanager prints two types of logs. One is the error log caused by the DDL-related binlog. The other is the info log where the table has been waiting for the field schame to update
1、Invalid historyRecord, because tableChanges should contain exactly 1 item.
2、Field [???] not found. Waiting for schema update.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the mysql_sync_database action and trace how the reported DDL binlog events and schema-update waits are handled. Reproduce the table alteration and insert sequence, then verify that schema changes are captured and the Flink checkpoint completes instead of remaining IN_PROGRESS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100