High deserialize time when doing shuffle read
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
Hi @marin-ma!
In one of our workloads we are seeing high deserialize time in columnar exchange node when shuffle data is read. This is hash based shuffle. The partitions were set to 1920 initially, we tried to decrease it to 1000 and then to 500 which decreases this time but still it's significant. What I observerd is that the number of batches become really huge after shuffle read annd number of rows per batch is very low. It improves with decreasing shuffle partitions but if we decrease it too much, we are not able to utilize the parallelism for other operators in the stage after shuffle read like sort, window, join and project with UDF. Wanted to get your input on this issue.
This is the screenshot with 1000 partitions.
This is with 500 partitions
The number of batches reduced from 148 million to 74 million but it's much higher than input batches which is 1.6 million.
I tried sort based shuffle also but didn't see any improvement.
Could you give some pointers that can help improve its performance?
### Gluten version
None
Contributor guide
Research direction
Start at the columnar exchange node's shuffle-read path, focusing on hash-based and sort-based shuffle behavior. Reproduce the workload and inspect deserialize time, batch counts, and rows per batch; done means reducing the excessive post-shuffle batches without preventing parallelism in downstream sort, window, join, and project operators.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100