flink schedule more compactions at once
Open
area:table-service
component:compaction
engine:flink
priority:low
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
flink CompactionPlanOperator schedule compaction at notifyCheckpointComplete that can only schedule one compaction every checkpoint.
when compaction rollback, compaction will later then compaction plan ,and cannot catch up with compaction plan create rate。eg: if compaction rollback 3 times, there will be 3 compaction plan pending schedule.
hudi table (RO) look like delay , and more compactions rollback , much longer hudi table dely, never catch up.
maybe we can make CompactionPlanOperator schedule more compactions at once
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3618
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.