apache / apache/gluten

[VL] Move `ColumnarBuildSideRelation`'s memory occupation to Spark off-heap

Open
#7,750 15 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

So far `ColumnarBuildSideRelation` is allocated on Spark JVM heap memory.

```
case class ColumnarBuildSideRelation(output: Seq[Attribute], batches: Array[Array[Byte]]) {
...
}
```

It appears that we can replace `batches: Array[Array[Byte]]` with an off-heap allocated container to move the memory usage to off-heap. ~~There should be a simple solution that doesn't require too much of refactor.~~ (see https://github.com/apache/incubator-gluten/pull/8127#discussion_r1895751275 about the edit)

This could avoid some of the heap OOM issues.

Contributor guide

Open the contributing guide

Research direction

Start by locating ColumnarBuildSideRelation and tracing how its batches field is allocated and consumed. Read the linked pull request discussion before choosing an off-heap container; done means batch storage moves off the JVM heap without changing relation behavior or introducing the heap OOM issues described.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.