[VL] Shuffle write size larger than vanilla spark
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
Conf:
spark.gluten.sql.columnar.shuffle.codec zstd
spark.gluten.sql.columnar.shuffle.compressionMode rowvector
spark.gluten.sql.columnar.shuffle.preferSpill true
vanilla spark

gluten

Through more specific logging, we found that this is related to the size of the ;arge buffer eventually used for compression in rowvector mode.
{[471711,41789,11]} means {[uncompressed buffer size, compressed buffer size, uncompressed/compressed]}
1. raw buffer size ~100k

2. raw buffer size ~200k

3. raw buffer size ~450k

Contributor guide
Research direction
Reproduce the shuffle comparison with the listed Gluten compression settings and compare vanilla Spark with Gluten. Start by investigating rowvector-mode compression and the large buffer sizes shown in the logs; done means identifying why Gluten writes more shuffle data and documenting or fixing the cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100