[CH] Thirteen ClickHouse test cases are permanently ignored after the Spark 3.3 removal
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Thirteen ClickHouse test cases are registered as ignored on every supported Spark version, and nothing runs them today.
They used to go through a helper `testSparkVersionLE33`, which ran the case on Spark 3.3 and ignored it on anything newer. Spark 3.3 was removed in #12902, so the `ignore` branch became the only one reachable. #12981 renamed the helper to `ignoreSpark33OnlyCase` and put the reason in its docstring, which makes the state visible but does not decide what to do about it.
What the thirteen cover:
| suite | cases |
|-|-|
| `GlutenClickHouseMergeTreeWriteSuite` | 2 |
| `GlutenClickHouseMergeTreeWriteOnHDFSSuite` | 2 |
| `GlutenClickHouseMergeTreeWriteOnHDFSWithRocksDBMetaSuite` | 2 |
| `GlutenClickHouseMergeTreeWriteOnS3Suite` | 2 |
| `GlutenClickHouseMergeTreeOptimizeSuite` | 1 |
| `GlutenClickHouseMergeTreePathBasedWriteSuite` | 1 |
| `GlutenClickhouseStringFunctionsSuite` | 1 (`base64`) |
| `GlutenClickHouseTPCHSaltNullParquetSuite` | 1 (posexplode, #1767) |
| `GlutenParquetFilterSuite` | 1 (a loop over queries) |
Nine of them are mergetree bucket-table writes, so that is where the coverage gap is concentrated.
### What needs deciding
Two directions, and it needs someone who owns the ClickHouse backend:
1. Make them pass on Spark 3.5 and drop `ignoreSpark33OnlyCase` along with them.
2. Delete the cases and accept that ClickHouse mergetree bucket writes have no test coverage.
Neither belongs in a version-check cleanup, which is why #12981 left them ignored.
### Related
A separate but similar question is open on the same PR: the two local `checkOperatorCount` helpers in `GlutenFunctionValidateSuite` and `GlutenClickHouseHiveTableSuite` had their whole body inside `if (spark33)`, so they now assert nothing while their call sites keep passing expected operator counts. Same choice applies: re-derive the counts on a supported version, or drop the helper and the counts.
cc @zzcclp @zhztheplayer
Contributor guide
Research direction
Start with the thirteen ignored cases in the named ClickHouse suites and the ignoreSpark33OnlyCase helper, then review the related checkOperatorCount helpers in GlutenFunctionValidateSuite and GlutenClickHouseHiveTableSuite. Decide with the ClickHouse backend owners whether to restore coverage on Spark 3.5 or remove the cases and assertions; done means the choice is implemented and no permanently ignored tests remain without an explicit rationale.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala
- Domain
- backend, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100