apache / apache/gluten

[VL] Shuffle write size larger than vanilla spark

Open
#3,267 25 comments 2 reactions 0 assignees View on GitHub
enhancement on track
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Conf:
spark.gluten.sql.columnar.shuffle.codec zstd
spark.gluten.sql.columnar.shuffle.compressionMode rowvector
spark.gluten.sql.columnar.shuffle.preferSpill true

vanilla spark
![image](https://github.com/oap-project/gluten/assets/142767528/4b062131-c17f-4399-80ff-f8ff1929a599)

gluten
![image](https://github.com/oap-project/gluten/assets/142767528/48ab9b10-c3cd-4c93-90d9-59b02573a497)

Through more specific logging, we found that this is related to the size of the ;arge buffer eventually used for compression in rowvector mode.
{[471711,41789,11]} means {[uncompressed buffer size, compressed buffer size, uncompressed/compressed]}
1. raw buffer size ~100k
![image](https://github.com/oap-project/gluten/assets/142767528/dd460a15-118d-4258-b487-35a29617317c)

2. raw buffer size ~200k
![image](https://github.com/oap-project/gluten/assets/142767528/db434852-5330-451e-bd00-3e834d0388ff)

3. raw buffer size ~450k
![image](https://github.com/oap-project/gluten/assets/142767528/3478101d-29ea-4b1b-b0ac-5b3d3a84bacc)

Contributor guide

Open the contributing guide

Research direction

Reproduce the shuffle comparison with the listed Gluten compression settings and compare vanilla Spark with Gluten. Start by investigating rowvector-mode compression and the large buffer sizes shown in the logs; done means identifying why Gluten writes more shuffle data and documenting or fixing the cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.