apache / apache/gluten

[VL] ColumnarPartialProjectExec may cause oom because it exports the entire batch

Open
#10,451 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

In https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarPartialProjectExec.scala, I notice that the entire batch is exported to Arrow format. Now Gluten use Arrow 15.0.0 and it doesn't implement StringView in java. So if there is any string column, it will have high memory pressure to gather the strings into a buffer.
I find that arrow 17.0.0 has introduced StringView implementation in java https://arrow.apache.org/release/17.0.0 https://github.com/apache/arrow/issues/40339. html So does gluten have any plan to use Arrow 17.0.0?
Or maybe we can implement a class named ArrowConverter and it controls the size of rows exported. But it will have a impact on the performance.

### Gluten version

None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.