[Bug] next_day rejects STRING and TIMESTAMP start dates that Spark accepts
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
`datafusion-spark`'s `next_day` declares
`Signature::exact(vec![DataType::Date32, DataType::Utf8], Volatility::Immutable)`
in `datafusion/spark/src/function/datetime/next_day.rs`.
Spark's `NextDay` extends `ImplicitCastInputTypes` with
`inputTypes = Seq(DateType, StringTypeWithCollation(...))`, so Spark casts the
first argument to `DATE` before evaluating. A `STRING` or a `TIMESTAMP` start
date is accepted. DataFusion raises a planning error instead.
This affects Spark 3.5.8 through 4.2.0 identically. The behavior has not
changed across those versions.
### To Reproduce
Spark SQL, verified against a live `pyspark==4.2.0` under both
`spark.sql.ansi.enabled=true` and `=false`:
```sql
SELECT next_day('2015-07-23', 'Mon');
-- 2015-07-27
SELECT next_day(TIMESTAMP '2015-07-23 12:12:12', 'Mon');
-- 2015-07-27
SELECT next_day('2015-07-23 12:12:12', 'Mon');
-- 2015-07-27
```
Spark's own golden file
`sql/core/src/test/resources/sql-tests/results/date.sql.out` asserts all three.
DataFusion SQL:
```sql
SELECT next_day('2015-07-23'::string, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Utf8View, Utf8View to the signature
-- Exact(Date32, Utf8) failed. No function matches the given name and argument
-- types 'next_day(Utf8View, Utf8View)'.
SELECT next_day('2015-07-23 12:12:12'::timestamp, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Timestamp(ns), Utf8View to the signature
-- Exact(Date32, Utf8) failed.
```
### Expected behavior
`next_day` should accept the argument types Spark accepts and cast them the way
Spark does.
### Additional context
Not fixed in the audit that surfaced this because it is a semantics decision
rather than a mechanical correction. Widening the signature raises questions
the maintainers should answer first:
1. Should `datafusion-spark` model Spark's `ImplicitCastInputTypes` generally,
or does it expect the caller to insert the casts? The answer applies to
every function in the crate, not just `next_day`, so a one-off widening here
would set an undeclared precedent.
2. Spark's STRING-to-DATE cast is itself ANSI dependent. `next_day('xx', 'Mon')`
raises `CAST_INVALID_INPUT` under ANSI and returns NULL otherwise. Any
widening has to route through a Spark-compatible cast, not
`arrow::compute::cast`.
3. `datafusion/sqllogictest/test_files/spark/datetime/next_day.slt` currently
asserts the coercion error as expected behavior. That assertion has to be
replaced, not merely extended.
The blocked cases are checked in as commented-out queries in
`datafusion/sqllogictest/test_files/spark/datetime/next_day.slt`, carrying the
Spark 4.2.0 results above. Uncommenting them is the contract for verifying a
fix.
Relevant file and line:
`datafusion/spark/src/function/datetime/next_day.rs`, `SparkNextDay::new`.
Surfaced by the audit-datafusion-spark-expression skill.
Contributor guide
Research direction
Read datafusion/spark/src/function/datetime/next_day.rs at SparkNextDay::new, then inspect datafusion/sqllogictest/test_files/spark/datetime/next_day.slt and the commented queries. First confirm the maintainers' policy for Spark-compatible input casting and ANSI behavior. Done means the agreed behavior is implemented and the blocked cases pass without retaining the old coercion-error assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100