apache / apache/datafusion

`generate_series` over timestamps accepts only nanosecond precision, with a misleading error

Open
#25,169 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`generate_series` / `range` over timestamps accept only nanosecond precision. Any other `TimeUnit` fails with an error that says the argument is not a timestamp, while naming a timestamp:

```sql
SELECT * FROM generate_series(
arrow_cast(TIMESTAMP '2024-01-01 00:00:00','Timestamp(Second, None)'),
arrow_cast(TIMESTAMP '2024-01-03 00:00:00','Timestamp(Second, None)'),
INTERVAL '1 day');

Error during planning: First argument must be a timestamp or NULL,
got Literal(TimestampSecond(1704067200, None), None)
```

The same query with `Timestamp(Nanosecond, …)` works.

### To Reproduce

The query above, on DataFusion 55.0.0 (`da89c7c85b`).

### Expected behavior

Either accept all four `TimeUnit`s (coercing to nanoseconds internally), or produce an error that says what is actually wrong — something like "expects nanosecond precision, got Timestamp(Second)".

The current message is actively misleading: it says the argument must be a timestamp and then prints a timestamp, which sends you looking in the wrong place. This matters more than it looks, because second and millisecond precision are common in Parquet files, so a `generate_series` over a column read from storage can fail while the same logic written with literals succeeds.

Found while adding timezone characterization tests in #25164.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the timestamp generate_series query from the issue on DataFusion 55.0.0, comparing nanosecond, second, millisecond, and microsecond precision. Trace the planning path for generate_series/range and the reported argument validation; done means supported time units work or the error accurately identifies the nanosecond requirement.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.