apache / apache/paimon

[Bug] Infinite loop after using online schema change to add a new column and update values

Open
#3,450 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.7.0

### Compute Engine

Flink 1.17.2

### Minimal reproduce step

I do this all in the local environment, please let me know if i missed some key steps, many thanks!
1. Docker build percona mysql 8.0
`docker run -d --name ps -e MYSQL_ROOT_USER=root -e MYSQL_ROOT_PASSWORD=123456 -p 3306:3306 percona/percona-server:8.0`
2. Add sample database `inventory` and table `customer_percona` with values
3. Install percona-toolkit 3.0.0
4. Setup and Run Flink job
`bin/flink run lib/paimon-flink-action-0.7-SNAPSHOT.jar mysql_sync_database --warehouse hdfs://localhost:8020/paimon/warehouse --database inventory --mysql_conf hostname=127.0.0.1 --mysql_conf port=3306 --mysql_conf username=root --mysql_conf password=123456 --mysql_conf database-name=inventory --mysql_conf server-time-zone=UTC --catalog_conf metastore=filesystem --table_conf bucket=4 --table_conf changelog-producer=input --table_conf sink.parallelism=4 --type_mapping to-nullable`
5. Proceed online schema change to add new column
`pt-online-schema-change --alter "add column bag_fee double" h=127.0.0.1,p=123456,u=root,P=3306,D=inventory,t=customer_percona --execute`
6. Update value in the new created column
``` UPDATE `inventory`.`customer_percona` SET `bag_fee` = '0' WHERE (`id` = '1');```

### What doesn't meet your expectations?

The writer keeps showing `Busy max 100%`
The flink-root-taskexecutor-xxxx.log keep waiting for schema update. `2024-05-31 12:25:45,596 INFO org.apache.paimon.flink.sink.cdc.CdcRecordUtils [] - Field bag_fee not found. Waiting for schema update.`

---
Before
![image (8)](https://github.com/apache/paimon/assets/79469011/0123a013-59f7-414e-8337-66ff96ebe867)

---
Online schema change
![image](https://github.com/apache/paimon/assets/79469011/1e457be2-8eeb-4456-965b-7779237b0843)

---
After updated value in the new created column
![image (9)](https://github.com/apache/paimon/assets/79469011/690d5883-fa75-42d3-a62d-d8687ac40e7b)
![image (10)](https://github.com/apache/paimon/assets/79469011/456f8bba-d93c-44c2-9b5a-a72883270733)

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Flink job from the issue using the listed MySQL and percona-toolkit commands. Inspect the CdcRecordUtils path that logs “Field bag_fee not found. Waiting for schema update.” Done means the online schema change and subsequent update no longer leave the writer at Busy max 100% or waiting indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.