UPSERTs are taking time
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Upsert is very slow and is taking 10 to 15 mins to load 678.0 KB into HUDI COW table. Not sure where the time is taken. Can some one please help me where is the issue?
**Environment Description**
* Hudi version : 0.10.1
* Spark version : Spark 3.1.2
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
Here is the Spark UI


Configs used:
"className": "org.apache.hudi",
"hoodie.table.name": tgt_tbl,
"hoodie.datasource.write.recordkey.field": "id",
"hoodie.datasource.write.precombine.field": "evnt_cent_tz",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.write.partitionpath.field": "year,month",
"hoodie.datasource.hive_sync.support_timestamp": "true",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.assume_date_partitioning": "false",
"hoodie.datasource.hive_sync.table": tgt_tbl,
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.mode":"hms",
"hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.upsert.shuffle.parallelism": 50,
"hoodie.delete.shuffle.parallelism": 50,
"hoodie.bulkinsert.sort.mode": "GLOBAL_SORT",
"hoodie.index.type": "BLOOM",
"hoodie.metadata.enable": "true",
"hoodie.metadata.index.bloom.filter.enable": "true",
"hoodie.metadata.index.bloom.filter.parallelism": 100,
"hoodie.metadata.index.bloom.filter.column.list": "id",
"hoodie.bloom.index.use.metadata": "true",
"hoodie.metadata.index.column.stats.enable": "true",
"hoodie.metadata.index.column.stats.column.list": "col1,col2,col3",
"hoodie.enable.data.skipping": "true"
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the attached Spark UI and the provided Hudi configuration, focusing on the upsert path for Hudi 0.10.1 with Spark 3.1.2 and S3. There are no source files or tests identified in the issue; done would be a reproducible explanation of where the 10–15 minutes is spent and any supported configuration or usage change that addresses it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100