[SUPPORT] Poor Upsert Performance on COW table due to indexing
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Hello, I am having performance issues when attempting to upsert data into a Hudi COW table. With the below specs it is taking longer than 4 hours to finish upserting (if it ever does finish). In the screenshots below, you can see that it is taking a long time doing the index scan. I have tried disabling hoodie.bloom.index.prune.by.ranges because our record key is random. I've also tried upserting using the "Simple" index type and did not see any performance improvements. Is there anything else I can do to improve the performance?

**Specs:**
Table Size: 13.6TB (compressed in S3)
Number of partitions: 1135 (hoodie.datasource.hive_sync.partition_fields=year,month)
Upsert dataset size: 68 million records, 6GB compressed
Index type: Default (Bloom)
Number of nodes: 30
Node type: r6g.8xlarge
Average record size: ~40 bytes (calculated by File Size/Num Records: 10MB/250000 records)
**Environment Description**
* Hudi version : 0.9.0
* Spark version : 2.4.8
* EMR version: 5.34.0
* Hive version : 2.38.0
* Hadoop version : Amazon 2.10.1
* Storage (HDFS/S3/GCS..) : S3
Contributor guide
No contributing guide indexed for this repository
Research direction
The report provides no repository files, tests, or code entry points to inspect. Start by reviewing the reported Hudi, Spark, Hadoop, S3, table-size, partition, index, and workload details, then identify the relevant indexing and upsert entry points; done would require a confirmed performance improvement or a documented configuration-based explanation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, spark
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100