[CH] findTopLevelSortMergeJoinTransform never matches on ClickHouse, so the migrated SPARK-37652 skew-join case is ignored on 3.4+
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
`ClickHouseAdaptiveQueryExecSuite` carries a Gluten rewrite of `SPARK-37652: optimize skewed join through union`. It was migrated from `gluten-ut/spark33` to the 3.4 / 3.5 / 4.0 / 4.1 modules in #12837, and it is committed there as `ignoreGluten` because it fails on ClickHouse 3.5.
The immediate failure is that `findTopLevelSortMergeJoinTransform` collects zero joins where the case expects two:
```
- Gluten - SPARK-37652: optimize skewed join through union *** FAILED ***
List() had size 0 instead of expected size 2 (ClickHouseAdaptiveQueryExecSuite.scala:1597)
```
This is a type mismatch rather than a plan-shape difference. The helper matches `org.apache.gluten.execution.SortMergeJoinExecTransformer`, which is the concrete Velox case class (`SortMergeJoinExecTransformer.scala:230`). ClickHouse plans these joins as `CHSortMergeJoinExecTransformer` or `CHShuffledHashJoinExecTransformer`, which extend `SortMergeJoinExecTransformerBase` (`SortMergeJoinExecTransformer.scala:37`) and `ShuffledHashJoinExecTransformerBase` respectively, and neither is a subtype of the Velox class. So on ClickHouse the helper returns an empty result for any query.
A backend-neutral type is available: `ColumnarShuffledJoin` (`JoinExecTransformer.scala:42`) declares `isSkewJoin` and is mixed into both `MergeJoinLikeExecTransformer` and `ShuffledHashJoinExecTransformerBase`, so `collect(plan) { case j: ColumnarShuffledJoin => j }` covers both backends and both join shapes.
That alone may not be enough to re-enable the case. The expected counts (2/2, 1/1, 3/0) are copied verbatim from spark33 and have never been checked against ClickHouse 3.5, so someone with a ClickHouse build needs to confirm or correct them.
Two related observations for whoever picks this up:
`findTopLevelSortMergeJoinTransform` has eight other call sites in the same suite, and they are all subject to the same mismatch on ClickHouse.
The neighbouring skew-join case in that suite already has its `checkSkewJoin(inner, 2, 1)` calls commented out (`ClickHouseAdaptiveQueryExecSuite.scala:707`), which suggests this area was parked earlier for what may be the same reason.
Fixing the helper touches shared test infrastructure and is out of scope for #12837, which is a pure test migration.
This description was written with AI assistance (Claude Code).
### Gluten version
main branch
Contributor guide
Research direction
Start with findTopLevelSortMergeJoinTransform and its eight call sites in ClickHouseAdaptiveQueryExecSuite.scala, then inspect the join transformer types in JoinExecTransformer.scala and SortMergeJoinExecTransformer.scala. Run the migrated SPARK-37652 case against ClickHouse 3.5 and compare the expected counts with the observed plans. Done means the helper covers the intended ClickHouse joins and the case is re-enabled with validated expectations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala, spark
- Domain
- backend, databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100