[VL] Array condition filtering is not supported
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
After setting spark.gluten.sql.complexType.scan.fallback.enabled=false in GlutenConfig. The below test is failed.
```
test("test SPARK-18053: ARRAY equality is broken") {
spark.range(10).select(array($"id").as("arr")).write.saveAsTable("array_tbl")
runQueryAndCompare("SELECT * FROM array_tbl where arr = ARRAY(1L)") { df =>
df.count() == 1
}
spark.sql(s"DROP TABLE IF EXISTS array_tbl").collect()
}
```
Expecting df.count to be 1, but it actually returns 10.
### Spark version
Spark-3.2.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
Results do not match for query:
Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
Timezone Env:
== Parsed Logical Plan ==
'Project [*]
+- 'Filter ('arr = 'ARRAY(1))
+- 'UnresolvedRelation [array_tbl], [], false
== Analyzed Logical Plan ==
arr: array
Project [arr#127]
+- Filter (arr#127 = cast(array(1) as array))
+- SubqueryAlias spark_catalog.default.array_tbl
+- Relation default.array_tbl[arr#127] parquet
== Optimized Logical Plan ==
Filter (isnotnull(arr#127) AND (arr#127 = [1]))
+- Relation default.array_tbl[arr#127] parquet
== Physical Plan ==
VeloxColumnarToRowExec
+- ^(2) FilterExecTransformer (isnotnull(arr#127) AND (arr#127 = [1]))
+- ^(2) NativeFileScan parquet default.array_tbl[arr#127] Batched: true, DataFilters: [isnotnull(arr#127), (arr#127 = [1])], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/root/jianzhen.wu/workspace/gluten/spark-warehouse/org.apache.glu..., PartitionFilters: [], PushedFilters: [IsNotNull(arr), EqualTo(arr,WrappedArray(1))], ReadSchema: struct>
== Results ==
== Results ==
!== Correct Answer - 1 == == Gluten Answer - 10 ==
struct<> struct<>
![ArrayBuffer(1)] [ArrayBuffer(0)]
! [ArrayBuffer(1)]
! [ArrayBuffer(2)]
! [ArrayBuffer(3)]
! [ArrayBuffer(4)]
! [ArrayBuffer(5)]
! [ArrayBuffer(6)]
! [ArrayBuffer(7)]
! [ArrayBuffer(8)]
! [ArrayBuffer(9)]
ScalaTestFailureLocation: org.apache.spark.sql.GlutenQueryTest$ at (GlutenQueryTest.scala:316)
org.scalatest.exceptions.TestFailedException:
Results do not match for query:
Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
Timezone Env:
== Parsed Logical Plan ==
'Project [*]
+- 'Filter ('arr = 'ARRAY(1))
+- 'UnresolvedRelation [array_tbl], [], false
== Analyzed Logical Plan ==
arr: array
Project [arr#127]
+- Filter (arr#127 = cast(array(1) as array))
+- SubqueryAlias spark_catalog.default.array_tbl
+- Relation default.array_tbl[arr#127] parquet
== Optimized Logical Plan ==
Filter (isnotnull(arr#127) AND (arr#127 = [1]))
+- Relation default.array_tbl[arr#127] parquet
== Physical Plan ==
VeloxColumnarToRowExec
+- ^(2) FilterExecTransformer (isnotnull(arr#127) AND (arr#127 = [1]))
+- ^(2) NativeFileScan parquet default.array_tbl[arr#127] Batched: true, DataFilters: [isnotnull(arr#127), (arr#127 = [1])], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/root/jianzhen.wu/workspace/gluten/spark-warehouse/org.apache.glu..., PartitionFilters: [], PushedFilters: [IsNotNull(arr), EqualTo(arr,WrappedArray(1))], ReadSchema: struct>
== Results ==
== Results ==
!== Correct Answer - 1 == == Gluten Answer - 10 ==
struct<> struct<>
![ArrayBuffer(1)] [ArrayBuffer(0)]
! [ArrayBuffer(1)]
! [ArrayBuffer(2)]
! [ArrayBuffer(3)]
! [ArrayBuffer(4)]
! [ArrayBuffer(5)]
! [ArrayBuffer(6)]
! [ArrayBuffer(7)]
! [ArrayBuffer(8)]
! [ArrayBuffer(9)]
at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at org.apache.spark.sql.GlutenQueryTest$.newAssertionFailedException(GlutenQueryTest.scala:302)
at org.scalatest.Assertions.fail(Assertions.scala:933)
at org.scalatest.Assertions.fail$(Assertions.scala:929)
at org.apache.spark.sql.GlutenQueryTest$.fail(GlutenQueryTest.scala:302)
at org.apache.spark.sql.GlutenQueryTest$.checkAnswer(GlutenQueryTest.scala:316)
at org.apache.spark.sql.GlutenQueryTest.checkAnswer(GlutenQueryTest.scala:212)
at org.apache.gluten.execution.WholeStageTransformerSuite.compareResultsAgainstVanillaSpark(WholeStageTransformerSuite.scala:309)
at org.apache.gluten.execution.WholeStageTransformerSuite.runQueryAndCompare(WholeStageTransformerSuite.scala:348)
at org.apache.gluten.execution.TestOperator.$anonfun$new$392(TestOperator.scala:2111)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:190)
at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:62)
at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:62)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
at org.scalatest.Suite.run(Suite.scala:1114)
at org.scalatest.Suite.run$(Suite.scala:1096)
at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:62)
at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:62)
at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
at scala.collection.immutable.List.foreach(List.scala:431)
at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
at org.scalatest.tools.Runner$.run(Runner.scala:798)
at org.scalatest.tools.Runner.run(Runner.scala)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:43)
at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:26)
```
Contributor guide
Research direction
Start with the failing test in TestOperator.scala at line 2111, named "test SPARK-18053: ARRAY equality is broken", and review the comparison path shown in the physical plan. Run the test with spark.gluten.sql.complexType.scan.fallback.enabled=false and verify that filtering arr = ARRAY(1L) returns only one row, matching vanilla Spark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100