apache / apache/gluten

[CH] findTopLevelSortMergeJoinTransform never matches on ClickHouse, so the migrated SPARK-37652 skew-join case is ignored on 3.4+

Open
#12,845 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

`ClickHouseAdaptiveQueryExecSuite` carries a Gluten rewrite of `SPARK-37652: optimize skewed join through union`. It was migrated from `gluten-ut/spark33` to the 3.4 / 3.5 / 4.0 / 4.1 modules in #12837, and it is committed there as `ignoreGluten` because it fails on ClickHouse 3.5.

The immediate failure is that `findTopLevelSortMergeJoinTransform` collects zero joins where the case expects two:

```
- Gluten - SPARK-37652: optimize skewed join through union *** FAILED ***
List() had size 0 instead of expected size 2 (ClickHouseAdaptiveQueryExecSuite.scala:1597)
```

This is a type mismatch rather than a plan-shape difference. The helper matches `org.apache.gluten.execution.SortMergeJoinExecTransformer`, which is the concrete Velox case class (`SortMergeJoinExecTransformer.scala:230`). ClickHouse plans these joins as `CHSortMergeJoinExecTransformer` or `CHShuffledHashJoinExecTransformer`, which extend `SortMergeJoinExecTransformerBase` (`SortMergeJoinExecTransformer.scala:37`) and `ShuffledHashJoinExecTransformerBase` respectively, and neither is a subtype of the Velox class. So on ClickHouse the helper returns an empty result for any query.

A backend-neutral type is available: `ColumnarShuffledJoin` (`JoinExecTransformer.scala:42`) declares `isSkewJoin` and is mixed into both `MergeJoinLikeExecTransformer` and `ShuffledHashJoinExecTransformerBase`, so `collect(plan) { case j: ColumnarShuffledJoin => j }` covers both backends and both join shapes.

That alone may not be enough to re-enable the case. The expected counts (2/2, 1/1, 3/0) are copied verbatim from spark33 and have never been checked against ClickHouse 3.5, so someone with a ClickHouse build needs to confirm or correct them.

Two related observations for whoever picks this up:

`findTopLevelSortMergeJoinTransform` has eight other call sites in the same suite, and they are all subject to the same mismatch on ClickHouse.

The neighbouring skew-join case in that suite already has its `checkSkewJoin(inner, 2, 1)` calls commented out (`ClickHouseAdaptiveQueryExecSuite.scala:707`), which suggests this area was parked earlier for what may be the same reason.

Fixing the helper touches shared test infrastructure and is out of scope for #12837, which is a pure test migration.

This description was written with AI assistance (Claude Code).

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start with findTopLevelSortMergeJoinTransform and its eight call sites in ClickHouseAdaptiveQueryExecSuite.scala, then inspect the join transformer types in JoinExecTransformer.scala and SortMergeJoinExecTransformer.scala. Run the migrated SPARK-37652 case against ClickHouse 3.5 and compare the expected counts with the observed plans. Done means the helper covers the intended ClickHouse joins and the case is re-enabled with validated expectations.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala, spark
Domain
backend, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.