apache / apache/datafusion-comet

SQLQueryTestSuite.ignoreList entries skip the Spark baseline, not just Comet

Open
#5,570 1 comment 0 reactions 1 assignee Claimed by @rich7420 View on GitHub
area:ci enhancement priority:medium test
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

`SQLQueryTestSuite.ignoreList` in the Spark test diffs carries two entries with no linked tracking issue:

```scala
// Comet: ORDER BY column has ties; row order is non-deterministic when
// running with high parallelism. Tracked for restoration once Comet
// produces stable ordering for these queries.
"replacing-missing-expression-with-alias.sql",
"in-set-operations.sql",
```

Present in `dev/diffs/4.1.3.diff` and `dev/diffs/4.2.0.diff`.

### Why this is worse than it looks

`ignoreList` is consulted in `createScalaTestCase` **before** any Comet-specific or test-type dispatch:

```scala
protected def createScalaTestCase(testCase: TestCase): Unit = {
if (ignoreList.exists(t =>
testCase.name.toLowerCase(Locale.ROOT).contains(t.toLowerCase(Locale.ROOT)))) {
ignore(testCase.name) { /* Do nothing */ }
} else testCase match {
```

so it registers a ScalaTest `ignore` unconditionally. The two files are therefore skipped in the `ENABLE_COMET=false` baseline as well, and in both the regular and analyzer variants: roughly 47 result queries plus three setup statements never run at all, in any mode. Nothing is being verified for them, not even Spark's own behaviour.

The same problem applied to `thetasketch.sql` / `tuplesketch.sql`; those were moved to the input-file `--SET spark.comet.enabled = false` header in #4950, which keeps every query running and compared against the golden file.

### What to do

1. Investigate the actual non-determinism. The stated cause is ORDER BY ties under parallelism; confirm whether it still reproduces on `main`, and on which queries.
2. If it does, replace the global `ignoreList` entries with the `--SET spark.comet.enabled = false` header used by the other opted-out files, so the Spark baseline and the golden comparison are preserved. Scope it to the affected queries if that is possible without regenerating goldens.
3. If it does not reproduce, remove the entries.

Surfaced in review of #4950 by @sunchao. Related: #5569 (stale exclusions whose tracking issues are already closed).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.