apache / apache/gluten

[VL] IllegalStateException thrown when accessing null values of map in partial project

Open
#11,330 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

The exception is as follows:
```
Reason: Operator::getOutput failed for [operator: ValueStream, plan node ID: 0]: Error during calling Java code from native code: java.lang.IllegalStateException: Value at index is null
at org.apache.gluten.shaded.org.apache.arrow.vector.BigIntVector.get(BigIntVector.java:105)
at org.apache.gluten.vectorized.ArrowWritableColumnVector$LongAccessor.getLong(ArrowWritableColumnVector.java:989)
at org.apache.gluten.vectorized.ArrowWritableColumnVector.getLong(ArrowWritableColumnVector.java:639)
at org.apache.spark.sql.vectorized.ColumnarArray.getLong(ColumnarArray.java:133)
at org.apache.spark.sql.catalyst.expressions.SpecializedGettersReader.read(SpecializedGettersReader.java:49)
at org.apache.spark.sql.vectorized.ColumnarArray.get(ColumnarArray.java:180)
at org.apache.spark.sql.catalyst.util.MapData.foreach(MapData.scala:38)
at org.apache.spark.sql.hive.HiveInspectors.$anonfun$wrapperFor$49(HiveInspectors.scala:433)
at org.apache.spark.sql.hive.HiveInspectors.$anonfun$withNullSafe$1(HiveInspectors.scala:262)
at org.apache.spark.sql.hive.HiveSimpleUDFEvaluator.setArg(hiveUDFEvaluators.scala:92)
at org.apache.spark.sql.hive.HiveSimpleUDF.$anonfun$eval$1(hiveUDFs.scala:67)
at org.apache.spark.sql.hive.HiveSimpleUDF.$anonfun$eval$1$adapted(hiveUDFs.scala:66)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.sql.hive.HiveSimpleUDF.eval(hiveUDFs.scala:66)
at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:158)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:76)
at org.apache.gluten.expression.InterpretedArrowProjection.apply(InterpretedArrowProjection.scala:33)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3(ColumnarPartialProjectExec.scala:233)
at org.apache.gluten.execution.ColumnarPartialProjectExec.$anonfun$getProjectedBatchArrow$3$adapted(ColumnarPartialProjectExec.scala:231)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at org.apache.gluten.execution.ColumnarPartialProjectExec.org$apache$gluten$execution$ColumnarPartialProjectExec$$getProjectedBatchArrow(ColumnarPartialProjectExec.scala:231)
at org.apache.gluten.execution.ColumnarPartialProjectExec$$anon$1.next(ColumnarPartialProjectExec.scala:176)
at org.apache.gluten.execution.ColumnarPartialProjectExec$$anon$1.next(ColumnarPartialProjectExec.scala:165)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
```

The reason is that [get](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java#L215) method of `ColumnarArray` doesn't handle null values(meaning that it will not check whether the value to get is a null value) and it is illegal to access a null value by [get](https://github.com/apache/arrow-java/blob/main/vector/src/main/java/org/apache/arrow/vector/BigIntVector.java#L98). So the exception is thrown when we try to access a null value of map.

### Gluten version

_No response_

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with ColumnarArray.java's get method and the ArrowWritableColumnVector accessors named in the stack trace, then trace how ColumnarPartialProjectExec.scala projects map values. Reproduce a partial project containing a map with null values and verify that accessing it no longer throws the reported IllegalStateException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.