add an option of the log file block size
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
on each hoodie log append, hdfs used will be increased with the length of the block(512M), not teh actual data length().
Consider in a scenario,I use many writers to append concurrently to a large number of files(bucket file),but each time I append only 10 bytes.
dfs used will be increased with the length of the block(512M),this will cause the datanode to report in-sufficient disk space on data write.
even though it related to HDFS, We should also have the option to modify the configuration.It helps reduce the rate of increase during the du.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4794
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Hudi's hoodie log append path and the configuration used for the HDFS log block size. Confirm how the fixed 512M block size is applied when appending small records, then identify the appropriate configurable setting and its existing configuration conventions. Done means concurrent small appends can use the configured block size without changing unrelated append behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100