apache / apache/gluten

[CH] Support `InSubqueryExec`

Open
#6,634 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

With following tables (with https://github.com/apache/incubator-gluten/pull/6570 enable)
```scala
case class TestData(key: Int, value: String)
case class TestData2(a: Int, b: Int)
case class TestData3(a: Int, b: Option[Int])
```
This query fails.
```sql
SELECT * FROM testData LEFT OUTER join testData2 ON key = a AND key NOT IN (select a from testData3) where value = '1';
```

```
Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.gluten.exception.GlutenNotSupportException: Not supported to map spark function name to substrait function name: key#24 IN subquery#41, class name: InSubqueryExec.
```

```
AdaptiveSparkPlan isFinalPlan=false
+- SortMergeJoin [key#24], [a#42], LeftOuter, NOT key#24 IN subquery#41
: +- Subquery subquery#41, [id=#490]
: +- AdaptiveSparkPlan isFinalPlan=false
: +- FileScan parquet default.testdata3[a#26] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct
:- Sort [key#24 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(key#24, 5), ENSURE_REQUIREMENTS, [plan_id=499]
: +- Filter (isnotnull(value#25) AND (value#25 = 1))
: +- FileScan parquet default.testdata[key#24,value#25] Batched: true, DataFilters: [isnotnull(value#25), (value#25 = 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [IsNotNull(value), EqualTo(value,1)], ReadSchema: struct
+- Sort [a#42 ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(a#42, 5), ENSURE_REQUIREMENTS, [plan_id=500]
+- BroadcastHashJoin [a#42], [a#26], LeftAnti, BuildRight, true
:- Filter isnotnull(a#42)
: +- FileScan parquet default.testdata2[a#42,b#43] Batched: true, DataFilters: [isnotnull(a#42)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [IsNotNull(a)], ReadSchema: struct
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),true), [plan_id=495]
+- FileScan parquet default.testdata3[a#26] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/data3/liangjiabiao/docker/local_gluten/spark-3.3.2-bin-hadoop3/s..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct

```

Contributor guide

Open the contributing guide

Research direction

Start at the InSubqueryExec handling identified by the GlutenNotSupportException and reproduce the supplied LEFT OUTER JOIN query with the listed test tables. Compare the execution plan and error before and after the change; done means the query runs successfully without the unsupported-function exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.