apache / apache/paimon

[Bug] Paimon mysql cdc When the mysql table structure changes, the task checkpoint cannot be submitted and is in the IN_PROGRESS state

Open
#3,364 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon-flink-1.17-0.8.jar
paimon-flink-action-0.8.0.jar
paimon-hive-connector-2.3-0.8.0.jar

### Compute Engine

flink version -> 1.17.2
hadoop -> 3.3.5

### Minimal reproduce step

1、create test table
CREATE TABLE `data_management`.`sql_template_info_paimon` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id',
`data_source_id` int(11) NOT NULL DEFAULT '1' COMMENT '数据源id, 1代表hive',
`table_name` varchar(100) NOT NULL COMMENT '当前操作的表名',
`chinese_table_name` varchar(1000) NOT NULL DEFAULT '' COMMENT '当前操作的中文表名',
`dimensionality` varchar(20) NOT NULL COMMENT '当前表的维度',
`table_in_tier` varchar(20) NOT NULL COMMENT '操作表所在层',
`is_partition_table` tinyint(1) NOT NULL COMMENT '是否为分区表',
`table_partition_type` varchar(100) NOT NULL COMMENT '表分区类型',
`table_importance_level` varchar(20) NOT NULL COMMENT '表重要等级',
`table_sensitivity_level` varchar(20) NOT NULL COMMENT '表敏感级别',
`partition_is_save_forever` tinyint(1) NOT NULL DEFAULT '0' COMMENT '分区是否永久保存',
`partition_keep_days` int(11) NOT NULL DEFAULT '7' COMMENT '分区保留天数,-1表示永久保存',
`update_frequency_level` varchar(20) NOT NULL COMMENT '更新频率',
`remark` longtext COMMENT '备注',
`old_table_name` varchar(200) DEFAULT NULL COMMENT '旧表名, 半角逗号隔开',
`define_label` varchar(100) DEFAULT NULL COMMENT '用户自定义标签,多个标签逗号隔开',
`heat` int(10) unsigned NOT NULL DEFAULT '0' COMMENT '热度',
`principal` varchar(60) NOT NULL COMMENT '当前表负责人',
`process_id` int(10) DEFAULT NULL COMMENT '工作流id',
`create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
`update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',
`special_date_save_partition` varchar(255) DEFAULT '' COMMENT '节假日保存分区信息'
PRIMARY KEY (`id`),
UNIQUE KEY `uniq_n` (`table_name`),
KEY `idx_process_id` (`process_id`)
) ENGINE=InnoDB AUTO_INCREMENT=68318 DEFAULT CHARSET=utf8 COMMENT='模板对应的基本信息表'

2、Synchronize tasks through "mysql_sync_database"
sudo ./bin/flink run --allowNonRestoredState /home/q/module/flink/flink-1.17.2/lib/paimon-flink-action-0.8.0.jar mysql_sync_database --warehouse viewfs://qunarcluster/user/tujiadev/home/data/paimon/warehouse --type_mapping tinyint1-not-bool,to-nullable,char-to-string,bigint-unsigned-to-bigint --database tujia_ods --mode combined --merge_shards false --table_prefix ods_ --table_suffix _paimon --mysql_conf hostname=xx --mysql_conf username=xx --mysql_conf password=xx --mysql_conf database-name='data_management' --mysql_conf port=3310 --mysql_conf server-id=6289-6298 --mysql_conf scan.incremental.snapshot.chunk.size=80960 --table_conf bucket=10 --table_conf sink.parallelism=1 --table_conf file.format=parquet --table_conf file.compression=snappy --table_conf scan.infer-parallelism=false --table_conf scan.parallelism=100 --including_tables 'mail_tasks' --catalog_conf metastore=hive --catalog_conf uri='thrift://l-hiveserver4.data.cn2:9083,thrift://l-hiveserver5.data.cn2:9083,thrift://l-hiveserver6.data.cn2:9083,thrift://l-hiveserver7.data.cn2:9083,thrift://l-hiveserver8.data.cn2:9083'

3、The sql submitted by mysql is as follows:

①、 "CREATE TABLE `data_management`.`_sql_template_info_paimon_new` (换行 `id` int(10) unsigned NOT NULL AUTO_INCREMENT COMMENT '自增id',换行 `data_source_id` int(11) NOT NULL DEFAULT '1' COMMENT '数据源id, 1代表hive',换行 `table_name` varchar(100) NOT NULL COMMENT '当前操作的表名',换行 `chinese_table_name` varchar(1000) NOT NULL DEFAULT '' COMMENT '当前操作的中文表名',换行 `dimensionality` varchar(20) NOT NULL COMMENT '当前表的维度',换行 `table_in_tier` varchar(20) NOT NULL COMMENT '操作表所在层',换行 `is_partition_table` tinyint(1) NOT NULL COMMENT '是否为分区表',换行 `table_partition_type` varchar(100) NOT NULL COMMENT '表分区类型',换行 `table_importance_level` varchar(20) NOT NULL COMMENT '表重要等级',换行 `table_sensitivity_level` varchar(20) NOT NULL COMMENT '表敏感级别',换行 `partition_is_save_forever` tinyint(1) NOT NULL DEFAULT '0' COMMENT '分区是否永久保存',换行 `partition_keep_days` int(11) NOT NULL DEFAULT '7' COMMENT '分区保留天数,-1表示永久保存',换行 `update_frequency_level` varchar(20) NOT NULL COMMENT '更新频率',换行 `remark` longtext COMMENT '备注',换行 `old_table_name` varchar(200) DEFAULT NULL COMMENT '旧表名, 半角逗号隔开',换行 `define_label` varchar(100) DEFAULT NULL COMMENT '用户自定义标签,多个标签逗号隔开',换行 `heat` int(10) unsigned NOT NULL DEFAULT '0' COMMENT '热度',换行 `principal` varchar(60) NOT NULL COMMENT '当前表负责人',换行 `process_id` int(10) DEFAULT NULL COMMENT '工作流id',换行 `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',换行 `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间',换行 `special_date_save_partition` varchar(255) DEFAULT '' COMMENT '节假日保存分区信息',换行 PRIMARY KEY (`id`),换行 UNIQUE KEY `uniq_n` (`table_name`),换行 KEY `idx_process_id` (`process_id`)换行) ENGINE=InnoDB AUTO_INCREMENT=68315 DEFAULT CHARSET=utf8 COMMENT='模板对应的基本信息表'",

②、 "ALTER TABLE `data_management`.`_sql_template_info_paimon_new` ADD COLUMN paimon_field VARCHAR(20) DEFAULT '' NOT NULL COMMENT '辨别'"

③、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_del` AFTER DELETE ON `data_management`.`sql_template_info_paimon` FOR EACH ROW DELETE IGNORE FROM `data_management`.`_sql_template_info_paimon_new` WHERE `data_management`.`_sql_template_info_paimon_new`.`id` <=> OLD.`id`"

④、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_upd` AFTER UPDATE ON `data_management`.`sql_template_info_paimon` FOR EACH ROW REPLACE INTO `data_management`.`_sql_template_info_paimon_new` (`id`, `data_source_id`, `table_name`, `chinese_table_name`, `dimensionality`, `table_in_tier`, `is_partition_table`, `table_partition_type`, `table_importance_level`, `table_sensitivity_level`, `partition_is_save_forever`, `partition_keep_days`, `update_frequency_level`, `remark`, `old_table_name`, `define_label`, `heat`, `principal`, `process_id`, `create_time`, `update_time`, `special_date_save_partition`) VALUES (NEW.`id`, NEW.`data_source_id`, NEW.`table_name`, NEW.`chinese_table_name`, NEW.`dimensionality`, NEW.`table_in_tier`, NEW.`is_partition_table`, NEW.`table_partition_type`, NEW.`table_importance_level`, NEW.`table_sensitivity_level`, NEW.`partition_is_save_forever`, NEW.`partition_keep_days`, NEW.`update_frequency_level`, NEW.`remark`, NEW.`old_table_name`, NEW.`define_label`, NEW.`heat`, NEW.`principal`, NEW.`process_id`, NEW.`create_time`, NEW.`update_time`, NEW.`special_date_save_partition`)"

⑤、 "CREATE DEFINER=`dba_inception_w`@`192.168.242.118` TRIGGER `pt_osc_data_management_sql_template_info_paimon_ins` AFTER INSERT ON `data_management`.`sql_template_info_paimon` FOR EACH ROW REPLACE INTO `data_management`.`_sql_template_info_paimon_new` (`id`, `data_source_id`, `table_name`, `chinese_table_name`, `dimensionality`, `table_in_tier`, `is_partition_table`, `table_partition_type`, `table_importance_level`, `table_sensitivity_level`, `partition_is_save_forever`, `partition_keep_days`, `update_frequency_level`, `remark`, `old_table_name`, `define_label`, `heat`, `principal`, `process_id`, `create_time`, `update_time`, `special_date_save_partition`) VALUES (NEW.`id`, NEW.`data_source_id`, NEW.`table_name`, NEW.`chinese_table_name`, NEW.`dimensionality`, NEW.`table_in_tier`, NEW.`is_partition_table`, NEW.`table_partition_type`, NEW.`table_importance_level`, NEW.`table_sensitivity_level`, NEW.`partition_is_save_forever`, NEW.`partition_keep_days`, NEW.`update_frequency_level`, NEW.`remark`, NEW.`old_table_name`, NEW.`define_label`, NEW.`heat`, NEW.`principal`, NEW.`process_id`, NEW.`create_time`, NEW.`update_time`, NEW.`special_date_save_partition`)"

⑥、 "ANALYZE TABLE `data_management`.`_sql_template_info_paimon_new` /* pt-online-schema-change */"
"RENAME TABLE `data_management`.`sql_template_info_paimon` TO `data_management`.`_sql_template_info_paimon_old`"

⑦、"DROP TABLE IF EXISTS `data_management`.`_sql_template_info_paimon_old`"

⑧、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_del`"
⑨、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_upd`"
⑩、"DROP TRIGGER IF EXISTS `data_management`.`pt_osc_data_management_sql_template_info_paimon_ins`"

4、Insert a random piece of data into this table

5、After the binlog consumes the current insert data, you can see that checkpoint is in the IN_PROGRESS state indefinitely

### What doesn't meet your expectations?

After the synchronized mysql table structure changes, the paimon table cannot capture the change, and most importantly, the entire task is stuck

### Anything else?

The taskmanager prints two types of logs. One is the error log caused by the DDL-related binlog. The other is the info log where the table has been waiting for the field schame to update
1、Invalid historyRecord, because tableChanges should contain exactly 1 item.
2、Field [???] not found. Waiting for schema update.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the mysql_sync_database action and trace how the reported DDL binlog events and schema-update waits are handled. Reproduce the table alteration and insert sequence, then verify that schema changes are captured and the Flink checkpoint completes instead of remaining IN_PROGRESS.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.