apache / apache/hudi

Support LogFileModTimeBasedCompactionStrategy

Open
#15,251 0 comments 0 reactions 0 assignees View on GitHub
area:table-service component:compaction from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

we need to support a new compaction strategy called LogFileModTimeBasedCompactionStrategy.

Using this strategy, we want to choose the file slice whose's earliest log file mod time for compaction. 

This will be similar to LogFileSizeBasedCompactionStrategy, just that instead of comparing total log files size for a given file slice, we will use earliest mod time for a given file slice. 

The goal is to compact some part of the whole change set (say, 20%) in one batch.
Compaction plan for a next batch should include incomplete operations from the previous plans.
Operations should be processed in order of earliest log file modification time.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4313
- Type: New Feature

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating LogFileSizeBasedCompactionStrategy and the existing compaction-strategy tests. Trace how file slices and incomplete operations are selected and ordered, then define the corresponding modification-time behavior. Done means the new strategy selects the intended portion of the change set, carries incomplete operations forward, and processes operations by earliest log-file modification time.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.