OversizedAllocationException in building arrow vectors
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
When Gluten uses Arrow vectors for variable-width columns, large string/binary data can hit Arrow's 2GB limit and fail:
```bash
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.shaded.org.apache.arrow.vector.util.OversizedAllocationException: Memory required for vector is (2147483648), which is overflow or more than max allowed (2147483647). You could consider using LargeVarCharVector/LargeVarBinaryVector for large strings/large bytes types
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.checkDataBufferSize(BaseVariableWidthVector.java:435)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.reallocDataBuffer(BaseVariableWidthVector.java:542)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.handleSafe(BaseVariableWidthVector.java:1304)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:1101)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setBytes(ArrowWritableColumnVector.java:1958)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setValueNullSafe(ArrowWritableColumnVector.java:1970)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$ArrowVectorWriter.write(ArrowWritableColumnVector.java:1450)
at org.apache.gluten.vectorized.ArrowWritableColumnVector.write(ArrowWritableColumnVector.java:776)
at org.apache.gluten.vectorized.ArrowColumnarRow.writeRow(ArrowColumnarRow.java:335)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:80)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:33)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3(ColumnarPartialProjectExec.scala:244)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3$adapted(ColumnarPartialProjectExec.scala:242)
```
### Gluten version
_No response_
Contributor guide
Research direction
Start in org.apache.gluten.vectorized.ArrowWritableColumnVector.java, especially StringWriter.setBytes and the vector-writing path shown in the stack trace. Trace how variable-width strings and binary values choose Arrow vector types, then reproduce the reported workload. Done means large values no longer fail at the 2GB allocation limit, with the relevant behavior covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100