[Bug] next_day cross-version divergence: ANSI default flip and error class change between Spark 3.5 and 4.x
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
`next_day` behaves differently across the Spark versions `datafusion-spark`
targets, and DataFusion's `.slt` files have no way to express a
version-specific expectation. This issue records the divergence so it can be
scheduled once that mechanism exists.
Companion to https://github.com/apache/datafusion/issues/23887.
### The divergence
Given a non-NULL start date and an unparseable `day_of_week`:
| Spark version | Default `spark.sql.ansi.enabled` | Default result | Error class when ANSI is on |
| ------------- | -------------------------------- | -------------- | --------------------------- |
| 3.5.8 | `false` | `NULL` | `_LEGACY_ERROR_TEMP_2000`, wrapped by `QueryExecutionErrors.ansiIllegalArgumentError` |
| 4.0.4 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 |
| 4.1.3 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 |
| 4.2.0 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 |
Two independent changes landed in Spark 4.0:
1. ANSI mode became the default, so the default result for the same query
flipped from `NULL` to an error.
2. `DateTimeUtils.getDayOfWeekFromString` changed from
`throw new IllegalArgumentException("Illegal input for day of week: ...")`
to `throw new SparkIllegalArgumentException(errorClass = "ILLEGAL_DAY_OF_WEEK", ...)`,
and `NextDay.nullSafeEval` changed from catching `IllegalArgumentException`
and re-wrapping to catching `SparkIllegalArgumentException` and rethrowing
unchanged.
A third 4.0 change is not currently observable: `inputTypes` moved from
`Seq(DateType, StringType)` to
`Seq(DateType, StringTypeWithCollation(supportsTrimCollation = true))`.
`datafusion-spark` has no collation support, so there is nothing to diverge
from yet, but a non-default collation on the `day_of_week` argument is a
divergence axis that exists only in Spark 4.x.
### Current state in DataFusion
Both modes are now implemented and tested behind
`datafusion.execution.enable_ansi_mode`, whose default is `false`. Anyone
comparing against a stock Spark 4.x will see DataFusion's default disagree with
Spark's default. The `.slt` file asserts Spark 4.2.0 behavior in each mode and
carries a comment naming the versions.
DataFusion does not model Spark error classes, so
`_LEGACY_ERROR_TEMP_2000` versus `ILLEGAL_DAY_OF_WEEK` is not represented at
all. `datafusion-spark` raises
`Execution error: Illegal input for day of week: `, matching Spark's
message text but not its error class or SQLSTATE.
### Additional context
Relevant files:
- `datafusion/spark/src/function/datetime/next_day.rs`
- `datafusion/sqllogictest/test_files/spark/datetime/next_day.slt`
- Spark: `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala`, `case class NextDay`
- Spark: `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala`, `getDayOfWeekFromString`
Surfaced by the audit-datafusion-spark-expression skill.
Contributor guide
Research direction
Read datafusion/spark/src/function/datetime/next_day.rs and datafusion/sqllogictest/test_files/spark/datetime/next_day.slt, then compare their ANSI-mode behavior with the Spark NextDay and DateTimeUtils references named in the issue. The work is ready only when version-specific expectations can be expressed and the Spark 3.5 and 4.x divergences are represented by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100