Revisiting doWriteOperation for preparing test data using java metadata table writer
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
[This method|https://github.com/apache/hudi/blob/master/hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java#L910] in HoodieTestTable is used to create commits and some test methods will create over 10 commits. Each call is taking 3-4 seconds locally for me so if we could cut this down to 1-2 seconds we would see a big testing performance improvement.
public HoodieCommitMetadata doWriteOperation(String commitTime, WriteOperationType operationType,
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5093
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5197
---
## Comments
09/Nov/22 13:15;xushiyan;The time-consuming part mainly comes from `org.apache.hudi.common.testutils.HoodieMetadataTestTable#doWriteOperation` which invokes spark metadata writer to update metadata table upon a new commit.
This is a necessary process. I don't think there is much room to optimize here. If we implement a java metadata writer, it may be faster but lose coverage around spark metadata writer, which is the major use case.
WDYT? [~guoyihua][~shivnarayan];;;
---
11/Nov/22 10:35;xushiyan;as discussed, we should tackle this with java writer implementation for data prep. will push this for future improvements.;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with doWriteOperation in hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java around line 910, then inspect HoodieMetadataTestTable#doWriteOperation and its Spark metadata-writer path. Compare the current 3–4 second local timing with the proposed Java metadata-writer approach, while checking the existing Spark-writer coverage; done means test-data commits are measurably faster without losing the intended coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100