apache / apache/datafusion

Support Spark-version-specific expectations in datafusion-spark sqllogictest files

Open
#23,887 1 comment 1 reaction 1 assignee Claimed by @andygrove View on GitHub
enhancement spark sqllogictest
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The sqllogictest files under `datafusion/sqllogictest/test_files/spark/` hard-code
their expected results, so each file encodes the behavior of exactly one Spark
version. There is currently no way to express that a function's behavior changed
between Spark versions.

This matters because Spark's built-in expressions do change across releases. A
few of the axes that move: ANSI mode became the default in Spark 4.0, collation
support was added to many string functions in 4.0, and individual expressions
have picked up new accepted input types or altered edge-case handling between
3.5 and 4.2.

Today the only options when we discover such a divergence are:

1. Assert one version's behavior and silently be wrong about the others.
2. Leave the case untested entirely.

Neither leaves a durable record, and neither lets CI catch a regression on the
version we did not pick.

Concretely, a test author who finds that `f(x)` returns `NULL` on Spark 3.5 and
throws on Spark 4.2 has nowhere to put that knowledge except a comment.

### Describe the solution you'd like

A way to scope a `.slt` file, or an individual query within one, to a Spark
version or version range.

There is prior art worth borrowing from. Apache DataFusion Comet solves the
analogous problem for its SQL file tests with a file-level directive:

```sql
-- MinSparkVersion: 3.5
```

which skips the whole file when running against an older Spark. See
`docs/source/contributor-guide/sql-file-tests.md` in apache/datafusion-comet.

Two shapes seem plausible here, and I do not have a strong preference yet:

**Option A: file-level directive.** Mirror Comet directly, e.g. a
`# MinSparkVersion: 4.0` comment directive honored by the sqllogictest runner.
Simple, but coarse: a single divergent query forces the whole file to be scoped,
or the divergent cases to be split into a separate file.

**Option B: per-query labels.** sqllogictest already supports `skipif ` /
`onlyif `. Exposing the target Spark version as a label would allow:

```sql
onlyif spark-4.2
query I
SELECT ...
----
42

onlyif spark-3.5
query I
SELECT ...
----
NULL
```

This is finer-grained and keeps both behaviors visible side by side at the test
site, which is where a reader wants them. It needs a decision on what the label
namespace looks like and how a range (rather than an exact version) is expressed.

Either option needs a notion of "which Spark version is this test suite currently
targeting", since there is no Spark process in the loop on the DataFusion side.
That target is a declared baseline rather than something detected at runtime.

### Describe alternatives you've considered

Keeping the status quo and recording divergences only in comments. This is what
we do implicitly today. It loses the ability to test the non-baseline behavior at
all, and the comments drift out of date with no signal.

### Additional context

This came up while designing an agent skill for auditing `datafusion-spark`
expression implementations against the Spark source. The audit compares a
function's behavior across several Spark releases, and cross-version divergences
are a routine finding. With no mechanism to express them, the audit can only
leave a comment and link here.

The related epic is #15914.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.