Support LogFileModTimeBasedCompactionStrategy
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
we need to support a new compaction strategy called LogFileModTimeBasedCompactionStrategy.
Using this strategy, we want to choose the file slice whose's earliest log file mod time for compaction.
This will be similar to LogFileSizeBasedCompactionStrategy, just that instead of comparing total log files size for a given file slice, we will use earliest mod time for a given file slice.
The goal is to compact some part of the whole change set (say, 20%) in one batch.
Compaction plan for a next batch should include incomplete operations from the previous plans.
Operations should be processed in order of earliest log file modification time.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4313
- Type: New Feature
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating LogFileSizeBasedCompactionStrategy and the existing compaction-strategy tests. Trace how file slices and incomplete operations are selected and ordered, then define the corresponding modification-time behavior. Done means the new strategy selects the intended portion of the change set, carries incomplete operations forward, and processes operations by earliest log-file modification time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100