apache / apache/hudi

add an option of the log file block size

Open
#15,410 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

on each hoodie log append, hdfs used will be increased with the length of the block(512M), not teh actual data length().
Consider in a scenario,I use many writers to append concurrently to a large number of files(bucket file),but each time I append only 10 bytes.
dfs used will be increased with the length of the block(512M),this will cause the datanode to report in-sufficient disk space on data write.
even though it related to HDFS, We should also have the option to modify the configuration.It helps reduce the rate of increase during the du.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4794
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Hudi's hoodie log append path and the configuration used for the HDFS log block size. Confirm how the fixed 512M block size is applied when appending small records, then identify the appropriate configurable setting and its existing configuration conventions. Done means concurrent small appends can use the configured block size without changing unrelated append behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.