[SUPPORT] Performance Tuning: Slow stages (Building Workload Profile & Getting Small files from partitions) during Hudi Writes
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Hi,
I am seeing some performance issues while upserting data especially in the below 2 jobs:
15 (SparkUpsertCommitActionExecutor)
17 (UpsertPartitioner)
Attached are some of the stats regarding the slow jobs/stages.
**Configurations used:**
--driver-memory 5G --executor-memory 10G --executor-cores 5 --num-executors 10
Upsert config parameters:
option(DataSourceWriteOptions.KEYGENERATOR_CLASS_OPT_KEY, "org.apache.hudi.keygen.ComplexKeyGenerator").
option("hoodie.upsert.shuffle.parallelism","2").
option("hoodie.insert.shuffle.parallelism","2").
option(HoodieCompactionConfig.PARQUET_SMALL_FILE_LIMIT_BYTES, 128 * 1024 * 1024).
option(HoodieStorageConfig.PARQUET_FILE_MAX_BYTES, 128 * 1024 * 1024).
option("hoodie.copyonwrite.record.size.estimate", "40")
Can you please guide how to approach tuning this performance problem? Let me know if you need any further details.
Below are some of the stats:

**Environment Description**
* Hudi version : 0.7.0
* Storage : GCS
* Running on Docker? (yes/no) : No
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.