apache / apache/datafusion

[Bug] next_day cross-version divergence: ANSI default flip and error class change between Spark 3.5 and 4.x

Open
#23,890 0 comments 0 reactions 0 assignees View on GitHub
bug spark
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`next_day` behaves differently across the Spark versions `datafusion-spark`
targets, and DataFusion's `.slt` files have no way to express a
version-specific expectation. This issue records the divergence so it can be
scheduled once that mechanism exists.

Companion to https://github.com/apache/datafusion/issues/23887.

### The divergence

Given a non-NULL start date and an unparseable `day_of_week`:

| Spark version | Default `spark.sql.ansi.enabled` | Default result | Error class when ANSI is on |
| ------------- | -------------------------------- | -------------- | --------------------------- |
| 3.5.8 | `false` | `NULL` | `_LEGACY_ERROR_TEMP_2000`, wrapped by `QueryExecutionErrors.ansiIllegalArgumentError` |
| 4.0.4 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 |
| 4.1.3 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 |
| 4.2.0 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 |

Two independent changes landed in Spark 4.0:

1. ANSI mode became the default, so the default result for the same query
flipped from `NULL` to an error.
2. `DateTimeUtils.getDayOfWeekFromString` changed from
`throw new IllegalArgumentException("Illegal input for day of week: ...")`
to `throw new SparkIllegalArgumentException(errorClass = "ILLEGAL_DAY_OF_WEEK", ...)`,
and `NextDay.nullSafeEval` changed from catching `IllegalArgumentException`
and re-wrapping to catching `SparkIllegalArgumentException` and rethrowing
unchanged.

A third 4.0 change is not currently observable: `inputTypes` moved from
`Seq(DateType, StringType)` to
`Seq(DateType, StringTypeWithCollation(supportsTrimCollation = true))`.
`datafusion-spark` has no collation support, so there is nothing to diverge
from yet, but a non-default collation on the `day_of_week` argument is a
divergence axis that exists only in Spark 4.x.

### Current state in DataFusion

Both modes are now implemented and tested behind
`datafusion.execution.enable_ansi_mode`, whose default is `false`. Anyone
comparing against a stock Spark 4.x will see DataFusion's default disagree with
Spark's default. The `.slt` file asserts Spark 4.2.0 behavior in each mode and
carries a comment naming the versions.

DataFusion does not model Spark error classes, so
`_LEGACY_ERROR_TEMP_2000` versus `ILLEGAL_DAY_OF_WEEK` is not represented at
all. `datafusion-spark` raises
`Execution error: Illegal input for day of week: `, matching Spark's
message text but not its error class or SQLSTATE.

### Additional context

Relevant files:

- `datafusion/spark/src/function/datetime/next_day.rs`
- `datafusion/sqllogictest/test_files/spark/datetime/next_day.slt`
- Spark: `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala`, `case class NextDay`
- Spark: `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala`, `getDayOfWeekFromString`

Surfaced by the audit-datafusion-spark-expression skill.

Contributor guide

Open the contributing guide

Research direction

Read datafusion/spark/src/function/datetime/next_day.rs and datafusion/sqllogictest/test_files/spark/datetime/next_day.slt, then compare their ANSI-mode behavior with the Spark NextDay and DateTimeUtils references named in the issue. The work is ready only when version-specific expectations can be expressed and the Spark 3.5 and 4.x divergences are represented by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.