apache / apache/hudi

[SUPPORT] How we can speed up individual file write(HoodieMergeHandle part)

Open
#10,997 5 comments 0 reactions 0 assignees View on GitHub
area:performance priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

We are using Spark 3.3 and Hudi 0.12.2.
I need your assistance in helping me to improve the `Doing partition and writing data` stage. For us, it looks to be the most time consuming. We are using `snappy` compression(the most lightweight from available as I know), file size is ~160mb, which is effectively 80-90 GB GZIP(default codec in Hudi for our workload). Files itself consist of 1.5-2M rows.
So our problem is that unfortunately due to partitioning + CDC nature, we must udpate a lot of files at peak hours, we have clustering to group rows together, but it's still thousands of files affected. 75th percentile of individual file overwrite(task in the `Doing partition and writing data` stage) takes ~40-60 seconds, it does not correlate to the number of rows updated inside(for 75th percentile it's < 100 rows changed in every file). Also - the payload class is almost default(minor changes which not affect performance IMO).
Q:
1. What are knobs we can play with?
We tried compression format(`snappy` looks to be the best among `zstd`- has memory leak in Spark 3.3 BTW and `gzip`)
Also we tried `hoodie.write.buffer.limit.bytes` - rising to 32MB, unfortunately no visible difference.
Is there any other?
2. Do you know some performance improvements in newer versions(0.12.3-0.14.1) regarding specifically file write(`MergeHandle`) task

**Environment Description**

* Hudi version : 0.12.2

* Spark version : 3.3.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the reported `HoodieMergeHandle` work in the `Doing partition and writing data` stage and the stated Spark 3.3.0, Hudi 0.12.2 environment. The issue does not name a source file, test, or specific change; it asks for performance knobs and comparisons with newer Hudi versions, so a concrete done condition is not defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.