[SUPPORT] How we can speed up individual file write(HoodieMergeHandle part)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
We are using Spark 3.3 and Hudi 0.12.2.
I need your assistance in helping me to improve the `Doing partition and writing data` stage. For us, it looks to be the most time consuming. We are using `snappy` compression(the most lightweight from available as I know), file size is ~160mb, which is effectively 80-90 GB GZIP(default codec in Hudi for our workload). Files itself consist of 1.5-2M rows.
So our problem is that unfortunately due to partitioning + CDC nature, we must udpate a lot of files at peak hours, we have clustering to group rows together, but it's still thousands of files affected. 75th percentile of individual file overwrite(task in the `Doing partition and writing data` stage) takes ~40-60 seconds, it does not correlate to the number of rows updated inside(for 75th percentile it's < 100 rows changed in every file). Also - the payload class is almost default(minor changes which not affect performance IMO).
Q:
1. What are knobs we can play with?
We tried compression format(`snappy` looks to be the best among `zstd`- has memory leak in Spark 3.3 BTW and `gzip`)
Also we tried `hoodie.write.buffer.limit.bytes` - rising to 32MB, unfortunately no visible difference.
Is there any other?
2. Do you know some performance improvements in newer versions(0.12.3-0.14.1) regarding specifically file write(`MergeHandle`) task
**Environment Description**
* Hudi version : 0.12.2
* Spark version : 3.3.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the reported `HoodieMergeHandle` work in the `Doing partition and writing data` stage and the stated Spark 3.3.0, Hudi 0.12.2 environment. The issue does not name a source file, test, or specific change; it asks for performance knobs and comparisons with newer Hudi versions, so a concrete done condition is not defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100