[VL][TEST] gluten-ut/spark34 matches the SQL query test list with contains, 3.5+ with equals
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### What
`GlutenSQLQueryTestSuite.createScalaTestCase` matches the supported list and the ignore list with `contains` on `gluten-ut/spark34`, and with `==` on `gluten-ut/spark35`, `spark40` and `spark41`. The exact-match versions carry a comment saying it is deliberate:
```scala
// gluten-ut/spark35/.../GlutenSQLQueryTestSuite.scala:459
// Modified for Gluten to use exact name matching.
!supportedList.exists(t => testCase.name.toLowerCase(Locale.ROOT) == t.toLowerCase(Locale.ROOT))
```
```scala
// gluten-ut/spark34/.../GlutenSQLQueryTestSuite.scala:315
!supportedList.exists(t => testCase.name.toLowerCase(Locale.ROOT).contains(t.toLowerCase(Locale.ROOT)))
```
spark34 never got that change.
### Why it matters
Two consequences, and the second one is the reason I am filing this.
**spark34 runs more than its list says.** A UDF `.sql` file expands into one test per UDF kind, named like `udf/udf-count.sql - Scala UDF`. Under `contains`, a short entry such as `"count.sql"` matches that name, so it silently enables all of that file's UDF variants. `VeloxSQLQueryTestSettings` on spark34 has nine such short entries: `count.sql`, `cross-join.sql`, `having.sql`, `inner-join.sql`, `join-empty-relation.sql`, `natural-join.sql`, `outer-join.sql`, `datetime-special.sql`, `parse-schema-string.sql`. I confirmed the effect by running the suite against Spark 3.4.4 with Velox: 222 tests run, and among them `udf/udf-count.sql - Scala UDF` and `udf/udf-count.sql - Regular Python UDF`, neither of which appears in the list in any form. On spark35 the same files do not run, because there the name has to match exactly.
**The two sides cannot be compared by set arithmetic.** Anyone auditing which `.sql` files run on which version, for example while checking what removing `gluten-ut/spark33` would drop (#12863, #12807), will diff the lists and get a wrong answer for spark34. I did exactly that and it produced a nine-file "gap" that turned out not to exist: those files already run on spark34 through the short entries.
### Two ways to fix it
Aligning spark34 with 3.5+ is the consistent option, but it is not free: the Regular Python UDF variants that run today on spark34 would stop, because the 3.5+ lists only enumerate `... - Scala UDF`. So it is a real coverage decision, not a mechanical change, which is why this is an issue rather than a patch.
1. Switch spark34 to `==` and enumerate whatever should keep running, accepting that the Python UDF variants go away there as they already have on 3.5+.
2. Keep `contains` on spark34 and instead document it, so the next audit does not compare the lists directly.
Either way the fix should include a check that the set of tests actually run does not change unintentionally; the suite runs locally against a Spark source tree via `-Dspark.test.home`, so a before/after diff of the run names is cheap.
Related: #12807, #12863, #12890.
Contributor guide
Research direction
Start in gluten-ut/spark34/.../GlutenSQLQueryTestSuite.scala and inspect VeloxSQLTestSettings entries that use short names. Run the suite against Spark 3.4.4 with -Dspark.test.home and record the test names before and after the chosen matching behavior. Done means the intended test set is explicit and the run-name diff shows no unintentional coverage change, or the contains behavior is documented for future audits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100