[CH] Support `InSubqueryExec`
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
With following tables (with https://github.com/apache/incubator-gluten/pull/6570 enable)
```scala
case class TestData(key: Int, value: String)
case class TestData2(a: Int, b: Int)
case class TestData3(a: Int, b: Option[Int])
```
This query fails.
```sql
SELECT * FROM testData LEFT OUTER join testData2 ON key = a AND key NOT IN (select a from testData3) where value = '1';
```
```
Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.gluten.exception.GlutenNotSupportException: Not supported to map spark function name to substrait function name: key#24 IN subquery#41, class name: InSubqueryExec.
```
```
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [key#24], [a#42], LeftOuter, NOT key#24 IN subquery#41
: +- Subquery subquery#41, [id=#490]
: +- AdaptiveSparkPlan isFinalPlan=false
: +- FileScan parquet default.testdata3[a#26] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct
:- Sort [key#24 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(key#24, 5), ENSURE_REQUIREMENTS, [plan_id=499]
: +- Filter (isnotnull(value#25) AND (value#25 = 1))
: +- FileScan parquet default.testdata[key#24,value#25] Batched: true, DataFilters: [isnotnull(value#25), (value#25 = 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [IsNotNull(value), EqualTo(value,1)], ReadSchema: struct
+- Sort [a#42 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(a#42, 5), ENSURE_REQUIREMENTS, [plan_id=500]
+- BroadcastHashJoin [a#42], [a#26], LeftAnti, BuildRight, true
:- Filter isnotnull(a#42)
: +- FileScan parquet default.testdata2[a#42,b#43] Batched: true, DataFilters: [isnotnull(a#42)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [IsNotNull(a)], ReadSchema: struct
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),true), [plan_id=495]
+- FileScan parquet default.testdata3[a#26] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct
```
Contributor guide
Research direction
Start at the InSubqueryExec handling identified by the GlutenNotSupportException and reproduce the supplied LEFT OUTER JOIN query with the listed test tables. Compare the execution plan and error before and after the change; done means the query runs successfully without the unsupported-function exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100