apache / apache/gluten

[VL] Poor Shuffle Read Perf in TPCH

Open
#4,750 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

### Backend
Velox
### Perf description
When switching from tpch 1T to tpch 3T, the Shuffle Read time of q21 will significantly increase, and the system's disk/network resources are not fully utilized.
In the DAG visualization, it can be seen that the time taken by the InputIterator in the 3T dataset is 60 times that of the 1T dataset.
![image](https://github.com/oap-project/gluten/assets/29010345/372c8102-ce74-4986-be42-6469a58a6c6a)
![image](https://github.com/oap-project/gluten/assets/29010345/15a55509-af67-4ffb-ace3-01e912c725f8)
In the Timeline, it can also be seen that the ShuffleRead time is unreasonable.
![image](https://github.com/oap-project/gluten/assets/29010345/55b6b0da-823d-4523-97d4-644255ac378e)
I observed the system's disk and network usage and found that the utilization is not high. I created a flame graph that includes sleep time, and it can be seen that the time taken by the LowCopyFileSegmentJniByteInputStream.read function has significantly increased.
1T:
![image](https://github.com/oap-project/gluten/assets/29010345/459edda4-0e14-4646-aade-83b2ff2e26fd)
3T:
![image](https://github.com/oap-project/gluten/assets/29010345/21c60dbf-6eec-4a5a-ab6c-54533b341f3d)
### Spark version
Spark-3.3.2
### Spark configurations
spark.executor.cores 4
spark.executor.memory 2G
spark.executor.memoryOverhead 1g
spark.memory.offHeap.size 10g
spark.gluten.sql.columnar.backend.velox.maxSpillFileSize 1073741824

Contributor guide

Open the contributing guide

Research direction

Start with the TPCH q21 comparison described in the issue, using the stated Spark 3.3.2 configuration and 1T versus 3T datasets. Profile InputIterator and LowCopyFileSegmentJniByteInputStream.read, then compare ShuffleRead time with disk and network utilization. Done means the 3T shuffle-read regression is explained and addressed or narrowed to a specific subsystem.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.