apache / apache/hudi

support EventTimeBasedCompactionStrategy

Open
#16,273 2 comments 0 reactions 0 assignees View on GitHub
area:table-service component:compaction from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The current compaction strategies are based on the logfile size, the number of logfile files, etc. The data time of the RO table generated by these strategies is uncontrollable. Hudi also has a DayBased strategy, but it relies on day based partition path and the time granularity is coarse.

The *EventTimeBasedCompactionStrategy* strategy can generate event time-friendly RO tables, whether it is day based partition or not. For example, the strategy can select all logfiles whose data time is before 3 am for compaction, so that the generated RO table data is before 3 am. If we just want to query data before 3 am, we can just query the RO table which is much faster.

With the strategy, I think we can expand the application scenarios of RO tables.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6979
- Type: New Feature

---

## Comments

30/Nov/23 16:53;shivnarayan;this will definitely be a good addition

 ;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing Hudi's existing compaction strategies, especially the DayBased strategy, and how they select logfiles for compaction. Define how event time and a cutoff such as 3 am should determine selection, then verify that the resulting RO table contains only data before the requested cutoff and supports the intended faster query scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.