Make sure Hoodie Max File Configs are respected for all Formats
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
This is an extension of HUDI-3709.
We need to revisit way we track the file sizes and avoid relying on HoodieWrapperFileSystem, since this method
* Is not reliable: file might be written lazily, therefore at FS level we won't know how much Writer has accumulated until it's closed (which is what was causing HUDI-3709)
* Since this method is quite low-level it has performance implications given that we're no intercepting low-level writes to increment the counters (stored globally on FS level)
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3715
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading HUDI-3709 and tracing the file-size tracking through HoodieWrapperFileSystem. Compare how max file configurations are applied across all formats, and consider the lazy-write and low-level performance concerns described here. Done means the configurations are respected without relying on the current filesystem-level accounting approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100