[VL] ColumnarPartialProjectExec may cause oom because it exports the entire batch
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
In https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/execution/ColumnarPartialProjectExec.scala, I notice that the entire batch is exported to Arrow format. Now Gluten use Arrow 15.0.0 and it doesn't implement StringView in java. So if there is any string column, it will have high memory pressure to gather the strings into a buffer.
I find that arrow 17.0.0 has introduced StringView implementation in java https://arrow.apache.org/release/17.0.0 https://github.com/apache/arrow/issues/40339. html So does gluten have any plan to use Arrow 17.0.0?
Or maybe we can implement a class named ArrowConverter and it controls the size of rows exported. But it will have a impact on the performance.
### Gluten version
None
Contributor guide
Assessment
This issue has not been assessed yet.