apache / apache/gluten

[VL] Support reading offHeap memory from ResourceProfile

Open
#10,099 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Stage-level adjustment is already supported https://github.com/apache/incubator-gluten/pull/8209. However, Velox is still [reading the off-heap memory settings from SparkConf](https://github.com/apache/incubator-gluten/blob/f1fc5697d086a5b4a518edba318f44b7c7bf0860/cpp/velox/compute/WholeStageResultIterator.cc#L485). Our team tried increasing the off-heap memory size using ResourceProfile in gluten, but Velox continues to use the earlier value from SparkConf instead of recognizing the updated setting.

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start with the ResourceProfile stage-level adjustment described in the issue and inspect WholeStageResultIterator.cc around the linked line where Velox reads off-heap settings from SparkConf. Trace how the updated ResourceProfile value reaches native execution instead of the earlier SparkConf value. Done means Velox uses the ResourceProfile off-heap setting for the affected stage.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, scala
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.