apache / apache/hudi

Revisiting doWriteOperation for preparing test data using java metadata table writer

Open
#15,516 1 comment 0 reactions 0 assignees View on GitHub
area:tests from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

[This method|https://github.com/apache/hudi/blob/master/hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java#L910] in HoodieTestTable is used to create commits and some test methods will create over 10 commits. Each call is taking 3-4 seconds locally for me so if we could cut this down to 1-2 seconds we would see a big testing performance improvement.
public HoodieCommitMetadata doWriteOperation(String commitTime, WriteOperationType operationType,

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5093
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-5197

---

## Comments

09/Nov/22 13:15;xushiyan;The time-consuming part mainly comes from `org.apache.hudi.common.testutils.HoodieMetadataTestTable#doWriteOperation` which invokes spark metadata writer to update metadata table upon a new commit.

This is a necessary process. I don't think there is much room to optimize here. If we implement a java metadata writer, it may be faster but lose coverage around spark metadata writer, which is the major use case.

WDYT? [~guoyihua][~shivnarayan];;;

---

11/Nov/22 10:35;xushiyan;as discussed, we should tackle this with java writer implementation for data prep. will push this for future improvements.;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with doWriteOperation in hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestTable.java around line 910, then inspect HoodieMetadataTestTable#doWriteOperation and its Spark metadata-writer path. Compare the current 3–4 second local timing with the proposed Java metadata-writer approach, while checking the existing Spark-writer coverage; done means test-data commits are measurably faster without losing the intended coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.