apache / apache/gluten

[VL] Limit Velox global memory manager's usage

Open
#6,960 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

It's observed that Velox backend uses more memory than we configured. Which is perhaps related to the untracked Velox global memory manager.

We should set a capacity according to Spark overhead size to [here](https://github.com/apache/incubator-gluten/blob/63144566d50a697daebdf2f3c569bddc44e398a5/cpp/velox/compute/VeloxBackend.cc#L130) to limit that memory manager's usage.

Contributor guide

Open the contributing guide

Research direction

Start at cpp/velox/compute/VeloxBackend.cc around line 130 and inspect how the Velox global memory manager is configured relative to Spark overhead size. Confirm the intended capacity and verify that the manager's usage is limited accordingly; the issue does not name a test, so identify the relevant validation path while reading the surrounding code.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, spark
Domain
backend, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.