apache / apache/hudi

[SUPPORT] Due to the untimely compaction, there are too many log files in the next compaction

Open
#9,019 4 comments 0 reactions 0 assignees View on GitHub
area:table-service priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

At present, hudi's compaction plan generation strategy is that if a FG appears in a certain plan, the FG will not enter the plan again;

For asynchronous compaction scenarios, if a plan is not executed in time, then if there is other data written to the FG later, the log file will continue to grow, which will cause the next plan to have a lot of log files, increasing the risk of compaction OOM;

Can we support that if a FG appears in a certain plan, the FG can still be generated in the plan?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Hudi's compaction plan generation and asynchronous compaction flow, focusing on how a file group already present in a plan is excluded from later plans. Reproduce the delayed-execution scenario described in the issue and define completion as allowing newly accumulated log files to be planned safely without introducing duplicate or conflicting compaction work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.