[VL] Bad spill performance
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
It's a known issue that spill has bad performance using Velox backend. Here is some data:
It also confirmed that Gluten only uses 1/2 Spark's memory size. With the same partition Gluten has no spill but Spark has.
And with partition size increases, sort based shuffle has better and more stable performance than hash based shuffle.
Contributor guide
Assessment
This issue has not been assessed yet.