apache / apache/datafusion-comet
SQLQueryTestSuite.ignoreList entries skip the Spark baseline, not just Comet
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
`SQLQueryTestSuite.ignoreList` in the Spark test diffs carries two entries with no linked tracking issue:
```scala
// Comet: ORDER BY column has ties; row order is non-deterministic when
// running with high parallelism. Tracked for restoration once Comet
// produces stable ordering for these queries.
"replacing-missing-expression-with-alias.sql",
"in-set-operations.sql",
```
Present in `dev/diffs/4.1.3.diff` and `dev/diffs/4.2.0.diff`.
### Why this is worse than it looks
`ignoreList` is consulted in `createScalaTestCase` **before** any Comet-specific or test-type dispatch:
```scala
protected def createScalaTestCase(testCase: TestCase): Unit = {
if (ignoreList.exists(t =>
testCase.name.toLowerCase(Locale.ROOT).contains(t.toLowerCase(Locale.ROOT)))) {
ignore(testCase.name) { /* Do nothing */ }
} else testCase match {
```
so it registers a ScalaTest `ignore` unconditionally. The two files are therefore skipped in the `ENABLE_COMET=false` baseline as well, and in both the regular and analyzer variants: roughly 47 result queries plus three setup statements never run at all, in any mode. Nothing is being verified for them, not even Spark's own behaviour.
The same problem applied to `thetasketch.sql` / `tuplesketch.sql`; those were moved to the input-file `--SET spark.comet.enabled = false` header in #4950, which keeps every query running and compared against the golden file.
### What to do
1. Investigate the actual non-determinism. The stated cause is ORDER BY ties under parallelism; confirm whether it still reproduces on `main`, and on which queries.
2. If it does, replace the global `ignoreList` entries with the `--SET spark.comet.enabled = false` header used by the other opted-out files, so the Spark baseline and the golden comparison are preserved. Scope it to the affected queries if that is possible without regenerating goldens.
3. If it does not reproduce, remove the entries.
Surfaced in review of #4950 by @sunchao. Related: #5569 (stale exclusions whose tracking issues are already closed).
Contributor guide
Assessment
This issue has not been assessed yet.