apache / apache/hudi

Make sure Hoodie Max File Configs are respected for all Formats

Open
#15,085 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

This is an extension of HUDI-3709.

We need to revisit way we track the file sizes and avoid relying on HoodieWrapperFileSystem, since this method

* Is not reliable: file might be written lazily, therefore at FS level we won't know how much Writer has accumulated until it's closed (which is what was causing HUDI-3709)
* Since this method is quite low-level it has performance implications given that we're no intercepting low-level writes to increment the counters (stored globally on FS level)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3715
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading HUDI-3709 and tracing the file-size tracking through HoodieWrapperFileSystem. Compare how max file configurations are applied across all formats, and consider the lazy-write and low-level performance concerns described here. Done means the configurations are respected without relying on the current filesystem-level accounting approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.