apache / apache/gluten

[CH] Thirteen ClickHouse test cases are permanently ignored after the Spark 3.3 removal

Open
#12,988 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Thirteen ClickHouse test cases are registered as ignored on every supported Spark version, and nothing runs them today.

They used to go through a helper `testSparkVersionLE33`, which ran the case on Spark 3.3 and ignored it on anything newer. Spark 3.3 was removed in #12902, so the `ignore` branch became the only one reachable. #12981 renamed the helper to `ignoreSpark33OnlyCase` and put the reason in its docstring, which makes the state visible but does not decide what to do about it.

What the thirteen cover:

| suite | cases |
|-|-|
| `GlutenClickHouseMergeTreeWriteSuite` | 2 |
| `GlutenClickHouseMergeTreeWriteOnHDFSSuite` | 2 |
| `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite` | 2 |
| `GlutenClickHouseMergeTreeWriteOnS3Suite` | 2 |
| `GlutenClickHouseMergeTreeOptimizeSuite` | 1 |
| `GlutenClickHouseMergeTreePathBasedWriteSuite` | 1 |
| `GlutenClickhouseStringFunctionsSuite` | 1 (`base64`) |
| `GlutenClickHouseTPCHSaltNullParquetSuite` | 1 (posexplode, #1767) |
| `GlutenParquetFilterSuite` | 1 (a loop over queries) |

Nine of them are mergetree bucket-table writes, so that is where the coverage gap is concentrated.

### What needs deciding

Two directions, and it needs someone who owns the ClickHouse backend:

1. Make them pass on Spark 3.5 and drop `ignoreSpark33OnlyCase` along with them.
2. Delete the cases and accept that ClickHouse mergetree bucket writes have no test coverage.

Neither belongs in a version-check cleanup, which is why #12981 left them ignored.

### Related

A separate but similar question is open on the same PR: the two local `checkOperatorCount` helpers in `GlutenFunctionValidateSuite` and `GlutenClickHouseHiveTableSuite` had their whole body inside `if (spark33)`, so they now assert nothing while their call sites keep passing expected operator counts. Same choice applies: re-derive the counts on a supported version, or drop the helper and the counts.

cc @zzcclp @zhztheplayer

Contributor guide

Open the contributing guide

Research direction

Start with the thirteen ignored cases in the named ClickHouse suites and the ignoreSpark33OnlyCase helper, then review the related checkOperatorCount helpers in GlutenFunctionValidateSuite and GlutenClickHouseHiveTableSuite. Decide with the ClickHouse backend owners whether to restore coverage on Spark 3.5 or remove the cases and assertions; done means the choice is implemented and no permanently ignored tests remain without an explicit rationale.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
backend, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.