[VL] IllegalStateException thrown when accessing null values of map in partial project
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
The exception is as follows:
```
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: java.lang.IllegalStateException: Value at index is null
at org.apache.gluten.shaded.org.apache.arrow.vector.BigIntVector.get(BigIntVector.java:105)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$LongAccessor.getLong(ArrowWritableColumnVector.java:989)
at org.apache.gluten.vectorized.ArrowWritableColumnVector.getLong(ArrowWritableColumnVector.java:639)
at org.apache.spark.sql.vectorized.ColumnarArray.getLong(ColumnarArray.java:133)
at org.apache.spark.sql.catalyst.expressions.SpecializedGettersReader.read(SpecializedGettersReader.java:49)
at org.apache.spark.sql.vectorized.ColumnarArray.get(ColumnarArray.java:180)
at org.apache.spark.sql.catalyst.util.MapData.foreach(MapData.scala:38)
at org.apache.spark.sql.hive.HiveInspectors.$anonfun$wrapperFor$49(HiveInspectors.scala:433)
at org.apache.spark.sql.hive.HiveInspectors.$anonfun$withNullSafe$1(HiveInspectors.scala:262)
at org.apache.spark.sql.hive.HiveSimpleUDFEvaluator.setArg(hiveUDFEvaluators.scala:92)
at org.apache.spark.sql.hive.HiveSimpleUDF.$anonfun$eval$1(hiveUDFs.scala:67)
at org.apache.spark.sql.hive.HiveSimpleUDF.$anonfun$eval$1$adapted(hiveUDFs.scala:66)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.sql.hive.HiveSimpleUDF.eval(hiveUDFs.scala:66)
at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:158)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:76)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:33)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3(ColumnarPartialProjectExec.scala:233)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3$adapted(ColumnarPartialProjectExec.scala:231)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at org.apache.gluten.execution.ColumnarPartialProjectExec.org$apache$gluten$execution$ColumnarPartialProjectExec$$getProjectedBatchArrow(ColumnarPartialProjectExec.scala:231)
at org.apache.gluten.execution.ColumnarPartialProjectExec$$anon$1.next(ColumnarPartialProjectExec.scala:176)
at org.apache.gluten.execution.ColumnarPartialProjectExec$$anon$1.next(ColumnarPartialProjectExec.scala:165)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
```
The reason is that [get](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java#L215) method of `ColumnarArray` doesn't handle null values(meaning that it will not check whether the value to get is a null value) and it is illegal to access a null value by [get](https://github.com/apache/arrow-java/blob/main/vector/src/main/java/org/apache/arrow/vector/BigIntVector.java#L98). So the exception is thrown when we try to access a null value of map.
### Gluten version
_No response_
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with ColumnarArray.java's get method and the ArrowWritableColumnVector accessors named in the stack trace, then trace how ColumnarPartialProjectExec.scala projects map values. Reproduce a partial project containing a map with null values and verify that accessing it no longer throws the reported IllegalStateException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100