apache / apache/datafusion

[Bug] next_day rejects STRING and TIMESTAMP start dates that Spark accepts

Open
#23,889 0 comments 0 reactions 0 assignees View on GitHub
bug spark
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`datafusion-spark`'s `next_day` declares
`Signature::exact(vec![DataType::Date32, DataType::Utf8], Volatility::Immutable)`
in `datafusion/spark/src/function/datetime/next_day.rs`.

Spark's `NextDay` extends `ImplicitCastInputTypes` with
`inputTypes = Seq(DateType, StringTypeWithCollation(...))`, so Spark casts the
first argument to `DATE` before evaluating. A `STRING` or a `TIMESTAMP` start
date is accepted. DataFusion raises a planning error instead.

This affects Spark 3.5.8 through 4.2.0 identically. The behavior has not
changed across those versions.

### To Reproduce

Spark SQL, verified against a live `pyspark==4.2.0` under both
`spark.sql.ansi.enabled=true` and `=false`:

```sql
SELECT next_day('2015-07-23', 'Mon');
-- 2015-07-27

SELECT next_day(TIMESTAMP '2015-07-23 12:12:12', 'Mon');
-- 2015-07-27

SELECT next_day('2015-07-23 12:12:12', 'Mon');
-- 2015-07-27
```

Spark's own golden file
`sql/core/src/test/resources/sql-tests/results/date.sql.out` asserts all three.

DataFusion SQL:

```sql
SELECT next_day('2015-07-23'::string, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Utf8View, Utf8View to the signature
-- Exact(Date32, Utf8) failed. No function matches the given name and argument
-- types 'next_day(Utf8View, Utf8View)'.

SELECT next_day('2015-07-23 12:12:12'::timestamp, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Timestamp(ns), Utf8View to the signature
-- Exact(Date32, Utf8) failed.
```

### Expected behavior

`next_day` should accept the argument types Spark accepts and cast them the way
Spark does.

### Additional context

Not fixed in the audit that surfaced this because it is a semantics decision
rather than a mechanical correction. Widening the signature raises questions
the maintainers should answer first:

1. Should `datafusion-spark` model Spark's `ImplicitCastInputTypes` generally,
or does it expect the caller to insert the casts? The answer applies to
every function in the crate, not just `next_day`, so a one-off widening here
would set an undeclared precedent.
2. Spark's STRING-to-DATE cast is itself ANSI dependent. `next_day('xx', 'Mon')`
raises `CAST_INVALID_INPUT` under ANSI and returns NULL otherwise. Any
widening has to route through a Spark-compatible cast, not
`arrow::compute::cast`.
3. `datafusion/sqllogictest/test_files/spark/datetime/next_day.slt` currently
asserts the coercion error as expected behavior. That assertion has to be
replaced, not merely extended.

The blocked cases are checked in as commented-out queries in
`datafusion/sqllogictest/test_files/spark/datetime/next_day.slt`, carrying the
Spark 4.2.0 results above. Uncommenting them is the contract for verifying a
fix.

Relevant file and line:
`datafusion/spark/src/function/datetime/next_day.rs`, `SparkNextDay::new`.

Surfaced by the audit-datafusion-spark-expression skill.

Contributor guide

Open the contributing guide

Research direction

Read datafusion/spark/src/function/datetime/next_day.rs at SparkNextDay::new, then inspect datafusion/sqllogictest/test_files/spark/datetime/next_day.slt and the commented queries. First confirm the maintainers' policy for Spark-compatible input casting and ANSI behavior. Done means the agreed behavior is implemented and the blocked cases pass without retaining the old coercion-error assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.