apache / apache/hudi

Job "Doing partition and writing data" performance regression from Hudi 0.9.0 to 0.14.0

Open
#16,971 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

User is running the same Hudi upsert application and same hudi configurations on Hudi versions 0.9.0 and 0.14.0 for performance benchmarking.
The results show that there is about a 2x performance regression on the job
{code:java}
Doing partition and writing data{code}
with ~2.5 mins on 0.9.0 and ~5 mins on 0.14.0. 
Is this a known issue on the performance regression and what is the cause of this regression?

Hudi config

{code:java}
upsert_hudi_config = {
"hoodie.table.name": "[table_name]",
"hoodie.database.name": "[database_name]",
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.NonpartitionedKeyGenerator",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.precombine.field": "[precombine_key]",
"hoodie.datasource.write.recordkey.field": "[record_key]",
"hoodie.datasource.write.table.name": "[table_name]",
"hoodie.index.type": "BLOOM",
"hoodie.metadata.enable": False,
"hoodie.upsert.shuffle.parallelism": 3,
}{code}
Data Characteristics

{code:java}
Table size: ~5GB uncompressed parquet data
Column count: 310 columns
High NULL density:
- Average NULLs per row: 217.74
- Min NULLs per row: 185
- Max NULLs per row: 230{code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9313
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the “Doing partition and writing data” job with the supplied upsert_hudi_config and the stated data characteristics on Hudi 0.9.0 and 0.14.0. Compare the benchmark and relevant execution stages to isolate the regression; done means identifying the cause and documenting or addressing the version-dependent performance change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.