Add support to skip larger log blocks with minor log compaction
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Is there a size threshold to exclude big log blocks? Why do log compaction on log blocks that are big enough? Thoughts
Good point. For initial version we want to target all the blocks. In the coming iterations I will include block sizes threshold as well. Current logic of AbstractHoodieLogRecordReader should be able to handle it as well.
For streaming workloads, this might be very heavy. So, we need to support this.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4728
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-3580
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with AbstractHoodieLogRecordReader and the linked HUDI-4728 improvement to understand the existing log-block compaction path. Clarify the size threshold and streaming-workload expectations, then verify that blocks above the threshold are skipped while smaller blocks retain current compaction behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100