apache / apache/gluten

OversizedAllocationException in building arrow vectors

Open
#12,447 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

When Gluten uses Arrow vectors for variable-width columns, large string/binary data can hit Arrow's 2GB limit and fail:
```bash
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: org.apache.gluten.shaded.org.apache.arrow.vector.util.OversizedAllocationException: Memory required for vector is (2147483648), which is overflow or more than max allowed (2147483647). You could consider using LargeVarCharVector/LargeVarBinaryVector for large strings/large bytes types
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.checkDataBufferSize(BaseVariableWidthVector.java:435)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.reallocDataBuffer(BaseVariableWidthVector.java:542)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.handleSafe(BaseVariableWidthVector.java:1304)
at org.apache.gluten.shaded.org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:1101)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setBytes(ArrowWritableColumnVector.java:1958)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$StringWriter.setValueNullSafe(ArrowWritableColumnVector.java:1970)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$ArrowVectorWriter.write(ArrowWritableColumnVector.java:1450)
at org.apache.gluten.vectorized.ArrowWritableColumnVector.write(ArrowWritableColumnVector.java:776)
at org.apache.gluten.vectorized.ArrowColumnarRow.writeRow(ArrowColumnarRow.java:335)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:80)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:33)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3(ColumnarPartialProjectExec.scala:244)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3$adapted(ColumnarPartialProjectExec.scala:242)
```

### Gluten version

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in org.apache.gluten.vectorized.ArrowWritableColumnVector.java, especially StringWriter.setBytes and the vector-writing path shown in the stack trace. Trace how variable-width strings and binary values choose Arrow vector types, then reproduce the reported workload. Done means large values no longer fail at the 2GB allocation limit, with the relevant behavior covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.