[VL] Limit Velox global memory manager's usage
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
It's observed that Velox backend uses more memory than we configured. Which is perhaps related to the untracked Velox global memory manager.
We should set a capacity according to Spark overhead size to [here](https://github.com/apache/incubator-gluten/blob/63144566d50a697daebdf2f3c569bddc44e398a5/cpp/velox/compute/VeloxBackend.cc#L130) to limit that memory manager's usage.
Contributor guide
Research direction
Start at cpp/velox/compute/VeloxBackend.cc around line 130 and inspect how the Velox global memory manager is configured relative to Spark overhead size. Confirm the intended capacity and verify that the manager's usage is limited accordingly; the issue does not name a test, so identify the relevant validation path while reading the surrounding code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, spark
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100