apache / apache/hudi

[SUPPORT] - Performance Variation in Hudi 0.14

Open
#11,481 7 comments 0 reactions 0 assignees View on GitHub
area:performance
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

**Describe the problem you faced**

We have a Glue 4.0 job to perform an upsert on a Hudi managed **COW** table. In some occasions, the Glue job runs in under 5 minutes, whereas in others it runs for up to 20 minutes. Moreover, we have noticed that, in those instances, the job is performing a count at `HoodieSparkSqlWriter.scala:1072` action for over 17 minutes; in other job runs this only takes around 1 minute.

Regarding some specifications for the table:

We have 3 partition fields:
* year : int
* month: int
* day : int

A precombine field:
* epoch: bigint

and 3 recordkey fields:
* node_id : string
* container_id : string
* container_label: string

You can see more about the table description here:

Screenshot 2024-06-21 at 13 31 19

We are also using a BLOOM type index and these are some other configurations that we are setting.

![Screenshot 2024-06-21 at 13 12 16](https://github.com/apache/hudi/assets/173461014/223d9b7b-54d5-4f48-803c-a8357425d748)

Could you please advise us on which actions we should take to bring down the execution time?

**Expected behavior**

We would like to understand why we are looking this variation in the execution times and advice on the actions needed

to prevent this behaviour.

**Environment Description**

* Glue version: 4

* Worker Type: G.2x

* Hudi version : 0.14.1

* Spark version : 3.3

* Max DPU Capacity: 120

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at HoodieSparkSqlWriter.scala around line 1072, where the report observes the long-running count action. Compare the faster and slower Glue 4.0 runs using the supplied Hudi, Spark, worker, DPU, S3, partition, key, and index details. Done means identifying the cause of the execution-time variation and documenting actions that could prevent it.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, scala
Domain
data-engineering, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.