Support custom compaction strategy to compact files partition in MDT aggressively
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
"files" partition in MDT is very critical for queries. so, we should compact it more frequently compared to other partitions. We need a custom compaction strategy for this.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5666
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1292
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with JIRA HUDI-5666 and the MDT compaction design; the issue does not name an implementation file, test, or entry point. Define how a custom strategy should identify the "files" partition and how its frequency differs from other partitions. Done means the strategy is implemented, documented, and covered by tests for aggressive compaction of that partition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100