[VL] Array with null value will fallback
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
```
val aai = Literal.create(Seq(Seq(1, 2, 3), null, Seq(4, 5)),
ArrayType(ArrayType(IntegerType, containsNull = false), containsNull = true))
16:32:17.693 WARN org.apache.spark.sql.execution.GlutenFallbackReporter: Validation failed for plan: Project[QueryId=0], due to: Native validation failed:
Optional[Validation failed due to exception caught at file:SubstraitToVeloxPlanValidator.cc line:1306 function:validate, thrown from file:SubstraitToVeloxExpr.cc line:400 function:literalsToArrayVector, reason:(0 vs. 0) there should be at least 1 value in list literal.].
```
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
"literal": {
"list": {
"values": [
{
"list": {
"values": [
{
"i32": 1
},
{
"i32": 2
},
{
"i32": 3
}
]
}
},
{
"null": {
"list": {
"type": {
"i32": {
"nullability": "NULLABILITY_REQUIRED"
}
},
"nullability": "NULLABILITY_NULLABLE"
}
}
},
{
"list": {
"values": [
{
"i32": 4
},
{
"i32": 5
}
]
}
}
]
}
}
```
Contributor guide
Research direction
Start with literalsToArrayVector in SubstraitToVeloxExpr.cc around line 400 and the validation call in SubstraitToVeloxPlanValidator.cc around line 1306. Reproduce the issue with the Literal.create array shown in the report, then verify that an array containing a null nested value no longer triggers native validation failure or fallback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100