apache / apache/hudi

Make user-provided copyOnWriteRecordSizeEstimate first precedence

Open
#15,492 1 comment 0 reactions 0 assignees View on GitHub
area:writer from-jira priority:high status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

For estimated avg record size
https://hudi.apache.org/docs/configurations/#hoodiecopyonwriterecordsizeestimate

which is used here
https://github.com/apache/hudi/blob/86a1efbff1300603a8180111eae117c7f9dbd8a5/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java#L372

Propose to respect user setting by following the precedence as below

1) if user sets a value, then use it as is
2) if user not setting it, infer from timeline commit metadata
3) if timeline is empty, use a default (current: 1024)

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5018
- Type: Improvement

---

## Comments

07/Nov/22 03:23;xichaomin;Hi [~xushiyan] , I can do this improvement, but I have a question, how to tell the avg value is set by user or is default?;;;

---

15/Nov/22 13:53;xushiyan;[~xichaomin] sure pls go ahead!;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java at the linked location and review the copyOnWriteRecordSizeEstimate configuration documentation. Verify the three requested cases—user value, timeline commit metadata, and the default value—using the project's relevant tests; done means user-provided settings take precedence.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.