[Bug] lastOptimizedCompaction in UniversalCompaction doesn't work as expected
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
master
### Compute Engine
Flink 1.18
### Minimal reproduce step
1. create a paimon table with optimization interval and enable debug log
```
CREATE TABLE IF NOT EXISTS table_a (
task_id varchar,
`value` bigint,
metric_name varchar,
PRIMARY KEY (`torrent_task_id`) NOT ENFORCED
) WITH (
'merge-engine' = 'partial-update',
'changelog-producer' = 'lookup',
'bucket' = '2',
'compaction.optimization-interval' = '24 h'
);
insert into table_a
select metric_name,`value`,task_id from kafka_table_b;
```
3. producing records to kafka
- produce records to kafka when the first checkpoint is triggered.
- don't produce any records for the next several checkpoints
- reproduce records to kafka
4.we'll see logs like that
```
DEBUG org.apache.paimon.operation.AbstractFileStoreWrite [] - Closing writer for partition org.apache.paimon.data.BinaryRow@9c67b85d, bucket 32. Writer's last modified identifier is 3300, while current commit identifier is 3302.
DEBUG org.apache.paimon.operation.AbstractFileStoreWrite [] - Creating writer for partition org.apache.paimon.data.BinaryRow@9c67b85d, bucket 32
DEBUG org.apache.paimon.mergetree.compact.UniversalCompaction [] - Universal compaction due to optimized compaction interval
```
### What doesn't meet your expectations?
Whenever MergeTreeWriter is recreated, as lastOptimizedCompaction is null in UniversalCompaction instance, full compaction will be triggered even if it is triggered before.
I think lastOptimizedCompaction may need to be persisted or retrieved from the file list, instead of being null all the time when UniversalCompaction instance is recreated.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with UniversalCompaction and the MergeTreeWriter recreation described in the report, then reproduce the behavior using the provided Paimon table and Flink 1.18 setup with debug logging. Trace how lastOptimizedCompaction is initialized across writer recreation and verify that repeated checkpoints do not trigger full compaction when the optimization interval was already satisfied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100