[VL] Enhancements to move more on-heap memory allocation to off-heap
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
This will be a series of efforts to move more on-heap memory allocation to off-heap against both Spark and Gluten's code, to reduce the risk of JVM heap OOMs, and to allow users to be able to configure a relatively smaller JVM heap size when Gluten is used.
Gluten:
- https://github.com/apache/incubator-gluten/issues/11133
Spark:
Tracked by https://issues.apache.org/jira/browse/SPARK-54089.
### Gluten version
None
Contributor guide
Research direction
The issue provides no files, tests, or entry points. Start by reading Gluten issue #11133 and Spark JIRA SPARK-54089 to identify the scoped allocation work; completion criteria are not defined here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100