support EventTimeBasedCompactionStrategy
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The current compaction strategies are based on the logfile size, the number of logfile files, etc. The data time of the RO table generated by these strategies is uncontrollable. Hudi also has a DayBased strategy, but it relies on day based partition path and the time granularity is coarse.
The *EventTimeBasedCompactionStrategy* strategy can generate event time-friendly RO tables, whether it is day based partition or not. For example, the strategy can select all logfiles whose data time is before 3 am for compaction, so that the generated RO table data is before 3 am. If we just want to query data before 3 am, we can just query the RO table which is much faster.
With the strategy, I think we can expand the application scenarios of RO tables.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6979
- Type: New Feature
---
## Comments
30/Nov/23 16:53;shivnarayan;this will definitely be a good addition
;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing Hudi's existing compaction strategies, especially the DayBased strategy, and how they select logfiles for compaction. Define how event time and a cutoff such as 3 am should determine selection, then verify that the resulting RO table contains only data before the requested cutoff and supports the intended faster query scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100