[Bug] flink compact_database cannot feel table schema change
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon version 1.0.1
### Compute Engine
flink version 1.17.1
### Minimal reproduce step
We observe inconsistent behavior when adding a new column to a Paimon table synced from MySQL via KafkaSyncDatabaseAction. The newly added column (bg) initially contains valid data, but values become null after database compaction and schema evolution.
1. Schema Evolution:
Add column bg (bigint) to MySQL source table
Execute schema sync on Paimon table:
ALTER TABLE T ADD COLUMNS (`bg` bigint COMMENT 'bg')
Verified: Newly inserted/updated data populates bg correctly in Paimon.
2. Periodic Compaction:
Run full database compaction using Flink job:
compact_database \
--warehouse hdfs:///user/paimon/warehouse_ods \
--including_databases 'database_1|database_2' \
--mode combined \
--table_conf continuous.discovery-interval=60s \
--table_conf snapshot.num-retained.min=10 \
--table_conf snapshot.time-retained=20m \
--table_conf full-compaction.delta-commits=30
3. Observed Issue:
Pre-compaction: bg column contains valid data
Post-compaction: bg column values become null in historical data
New writes continue to populate bg until next compaction
4. When I restarted the Compaction Flink job, this issue disappeared
### What doesn't meet your expectations?
Data in the newly added column (bg) should persist through compaction operations and remain visible in all snapshots.
### Anything else?
This is the phenomenon I have observed:

Is this my discovery in the source code caused by this reason?:




### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
The named entry points are KafkaSyncDatabaseAction and the compact_database Flink job; start by reproducing the schema-addition and compaction sequence on Paimon 1.0.1. Done means the bg values remain present in historical data after compaction and across snapshots.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100