Avoid 0 record parquet files
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
There are occasions where hudi could produce 0 record files.
For eg,
a. entire set of records deleted in log block and due to small file handling, a new parquet is created with HoodieMergeHandle.
b. During compaction, again there are chances that hudi might produce 0 record parquet files.
We need to avoid such files if possible.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7780
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading HoodieMergeHandle and the compaction paths mentioned in the issue. Trace the cases where all records are deleted or compaction produces no records, then determine how zero-record Parquet files are created. Done means those scenarios avoid producing such files, with coverage for both small-file handling and compaction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100