apache / apache/gluten

[VL] Optimize shuffle writer memory usage

Open
#3,444 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
3d 17h
Merged PRs (30d)
86

Description

### Description

A simple task with scan + filter + shuffle writer has 45MB of peek memory usage, assume scan and filter belong to computation part, shuffle writer belong to shuffle part, after add detailed log, we find computation part consume 25MB, shuffle part consume 20MB.

I'm not start but I think shuffle part's memory usage could be optimized.

![image](https://github.com/oap-project/gluten/assets/34979747/b9770453-b1c8-41e4-ba96-ea85b813cc76)

set spark.gluten.memory.reservationBlockSize=1MB to minimize reservation.

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Reproduce the scan, filter, and shuffle-writer task with spark.gluten.memory.reservationBlockSize=1MB, then use detailed memory logging to isolate the shuffle part's reservation; done means demonstrating lower shuffle memory usage without changing the computation measurement.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, distributed-systems, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.