apache / apache/paimon

[Bug] flink compact_database cannot feel table schema change

Open
#5,791 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon version 1.0.1

### Compute Engine

flink version 1.17.1

### Minimal reproduce step

We observe inconsistent behavior when adding a new column to a Paimon table synced from MySQL via KafkaSyncDatabaseAction. The newly added column (bg) initially contains valid data, but values become null after database compaction and schema evolution.

1. Schema Evolution:
Add column bg (bigint) to MySQL source table
Execute schema sync on Paimon table:
ALTER TABLE T ADD COLUMNS (`bg` bigint COMMENT 'bg')
Verified: Newly inserted/updated data populates bg correctly in Paimon.

2. Periodic Compaction:
Run full database compaction using Flink job:

compact_database \
--warehouse hdfs:///user/paimon/warehouse_ods \
--including_databases 'database_1|database_2' \
--mode combined \
--table_conf continuous.discovery-interval=60s \
--table_conf snapshot.num-retained.min=10 \
--table_conf snapshot.time-retained=20m \
--table_conf full-compaction.delta-commits=30

3. Observed Issue:
Pre-compaction: bg column contains valid data
Post-compaction: bg column values become null in historical data
New writes continue to populate bg until next compaction

4. When I restarted the Compaction Flink job, this issue disappeared

### What doesn't meet your expectations?

Data in the newly added column (bg) should persist through compaction operations and remain visible in all snapshots.

### Anything else?

This is the phenomenon I have observed:
![Image](https://github.com/user-attachments/assets/057d6f8a-bc00-4519-a8eb-f4daa92e98e5)

Is this my discovery in the source code caused by this reason?:

![Image](https://github.com/user-attachments/assets/de6b8f09-2d0d-45fb-b6eb-7c45ca904d9c)

![Image](https://github.com/user-attachments/assets/19ff66d2-bcd9-421a-8c3b-c979487e4ab9)

![Image](https://github.com/user-attachments/assets/bb75350e-5bd5-4136-beda-716bf8cf634e)

![Image](https://github.com/user-attachments/assets/ec69def2-1bae-4602-8b39-603e90551e83)

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The named entry points are KafkaSyncDatabaseAction and the compact_database Flink job; start by reproducing the schema-addition and compaction sequence on Paimon 1.0.1. Done means the bg values remain present in historical data after compaction and across snapshots.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.