[SUPPORT] Due to the untimely compaction, there are too many log files in the next compaction
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
At present, hudi's compaction plan generation strategy is that if a FG appears in a certain plan, the FG will not enter the plan again;
For asynchronous compaction scenarios, if a plan is not executed in time, then if there is other data written to the FG later, the log file will continue to grow, which will cause the next plan to have a lot of log files, increasing the risk of compaction OOM;
Can we support that if a FG appears in a certain plan, the FG can still be generated in the plan?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Hudi's compaction plan generation and asynchronous compaction flow, focusing on how a file group already present in a plan is excluded from later plans. Reproduce the delayed-execution scenario described in the issue and define completion as allowing newly accumulated log files to be planned safely without introducing duplicate or conflicting compaction work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100