[VL] Move `ColumnarBuildSideRelation`'s memory occupation to Spark off-heap
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
So far `ColumnarBuildSideRelation` is allocated on Spark JVM heap memory.
```
case class ColumnarBuildSideRelation(output: Seq[Attribute], batches: Array[Array[Byte]]) {
...
}
```
It appears that we can replace `batches: Array[Array[Byte]]` with an off-heap allocated container to move the memory usage to off-heap. ~~There should be a simple solution that doesn't require too much of refactor.~~ (see https://github.com/apache/incubator-gluten/pull/8127#discussion_r1895751275 about the edit)
This could avoid some of the heap OOM issues.
Contributor guide
Research direction
Start by locating ColumnarBuildSideRelation and tracing how its batches field is allocated and consumed. Read the linked pull request discussion before choosing an off-heap container; done means batch storage moves off the JVM heap without changing relation behavior or introducing the heap OOM issues described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100