apache / apache/gluten

High deserialize time when doing shuffle read

Open
#10,214 24 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Hi @marin-ma!
In one of our workloads we are seeing high deserialize time in columnar exchange node when shuffle data is read. This is hash based shuffle. The partitions were set to 1920 initially, we tried to decrease it to 1000 and then to 500 which decreases this time but still it's significant. What I observerd is that the number of batches become really huge after shuffle read annd number of rows per batch is very low. It improves with decreasing shuffle partitions but if we decrease it too much, we are not able to utilize the parallelism for other operators in the stage after shuffle read like sort, window, join and project with UDF. Wanted to get your input on this issue.

This is the screenshot with 1000 partitions.
Image

This is with 500 partitions

Image

The number of batches reduced from 148 million to 74 million but it's much higher than input batches which is 1.6 million.
I tried sort based shuffle also but didn't see any improvement.
Could you give some pointers that can help improve its performance?

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start at the columnar exchange node's shuffle-read path, focusing on hash-based and sort-based shuffle behavior. Reproduce the workload and inspect deserialize time, batch counts, and rows per batch; done means reducing the excessive post-shuffle batches without preventing parallelism in downstream sort, window, join, and project operators.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.