Make user-provided copyOnWriteRecordSizeEstimate first precedence
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
For estimated avg record size
https://hudi.apache.org/docs/configurations/#hoodiecopyonwriterecordsizeestimate
which is used here
https://github.com/apache/hudi/blob/86a1efbff1300603a8180111eae117c7f9dbd8a5/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java#L372
Propose to respect user setting by following the precedence as below
1) if user sets a value, then use it as is
2) if user not setting it, infer from timeline commit metadata
3) if timeline is empty, use a default (current: 1024)
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5018
- Type: Improvement
---
## Comments
07/Nov/22 03:23;xichaomin;Hi [~xushiyan] , I can do this improvement, but I have a question, how to tell the avg value is set by user or is default?;;;
---
15/Nov/22 13:53;xushiyan;[~xichaomin] sure pls go ahead!;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java at the linked location and review the copyOnWriteRecordSizeEstimate configuration documentation. Verify the three requested cases—user value, timeline commit metadata, and the default value—using the project's relevant tests; done means user-provided settings take precedence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100