apache / apache/gluten

[VL] Enhancements to move more on-heap memory allocation to off-heap

Open
#11,169 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

This will be a series of efforts to move more on-heap memory allocation to off-heap against both Spark and Gluten's code, to reduce the risk of JVM heap OOMs, and to allow users to be able to configure a relatively smaller JVM heap size when Gluten is used.

Gluten:

- https://github.com/apache/incubator-gluten/issues/11133

Spark:

Tracked by https://issues.apache.org/jira/browse/SPARK-54089.

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

The issue provides no files, tests, or entry points. Start by reading Gluten issue #11133 and Spark JIRA SPARK-54089 to identify the scoped allocation work; completion criteria are not defined here.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.