apache / apache/hudi

[SUPPORT] Poor Upsert Performance on COW table due to indexing

Open
#6,687 5 comments 0 reactions 0 assignees View on GitHub
area:index area:performance priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hello, I am having performance issues when attempting to upsert data into a Hudi COW table. With the below specs it is taking longer than 4 hours to finish upserting (if it ever does finish). In the screenshots below, you can see that it is taking a long time doing the index scan. I have tried disabling hoodie.bloom.index.prune.by.ranges because our record key is random. I've also tried upserting using the "Simple" index type and did not see any performance improvements. Is there anything else I can do to improve the performance?

image
image

**Specs:**
Table Size: 13.6TB (compressed in S3)
Number of partitions: 1135 (hoodie.datasource.hive_sync.partition_fields=year,month)
Upsert dataset size: 68 million records, 6GB compressed
Index type: Default (Bloom)
Number of nodes: 30
Node type: r6g.8xlarge
Average record size: ~40 bytes (calculated by File Size/Num Records: 10MB/250000 records)

**Environment Description**

* Hudi version : 0.9.0
* Spark version : 2.4.8
* EMR version: 5.34.0
* Hive version : 2.38.0
* Hadoop version : Amazon 2.10.1
* Storage (HDFS/S3/GCS..) : S3

Contributor guide

No contributing guide indexed for this repository

Research direction

The report provides no repository files, tests, or code entry points to inspect. Start by reviewing the reported Hudi, Spark, Hadoop, S3, table-size, partition, index, and workload details, then identify the relevant indexing and upsert entry points; done would require a confirmed performance improvement or a documented configuration-based explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, spark
Domain
data-engineering, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.