[VL] Optimize shuffle writer memory usage
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 86
Description
### Description
A simple task with scan + filter + shuffle writer has 45MB of peek memory usage, assume scan and filter belong to computation part, shuffle writer belong to shuffle part, after add detailed log, we find computation part consume 25MB, shuffle part consume 20MB.
I'm not start but I think shuffle part's memory usage could be optimized.

set spark.gluten.memory.reservationBlockSize=1MB to minimize reservation.
Contributor guide
Research direction
No source file or test is named. Reproduce the scan, filter, and shuffle-writer task with spark.gluten.memory.reservationBlockSize=1MB, then use detailed memory logging to isolate the shuffle part's reservation; done means demonstrating lower shuffle memory usage without changing the computation measurement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100