[CORE] Investigate: Heap shrinking to share on-heap memory for off-heap use
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
(ONLY A PATHFINDING NOW)
Based on accessible documents, there might be a way to do real memory sharing between on-heap and off-heap in Java so probably in Gluten, by shrinking on-heap memory dynamically and then increase the off-heap memory accordingly.
Preconditions:
* (Required) `-XX:MaxHeapFreeRatio` / `-XX:MinHeapFreeRatio` in G1 GC + HotSpot JVM (https://www.oracle.com/java/technologies/javase/vmoptions-jsp.html)
* (Optional?) \>= Java 12? ([JEP-346](https://openjdk.org/jeps/346))
* (Optional?) `-Xmaxf` / `-Xminf` in HotSpot JVM (https://docs.oracle.com/cd/E19683-01/806-7930/vmoptions-chapter/index.html)
This could be an enhancement on https://github.com/apache/incubator-gluten/issues/5438 which is easily causing OOM now.
Contributor guide
Research direction
Start with the linked HotSpot VM options, JEP-346, and related issue #5438; verify the G1/HotSpot prerequisites and whether the proposed heap and off-heap behavior is feasible. Done means a documented feasibility result and a concrete next step, since this issue is explicitly pathfinding rather than an implementation task.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100