apache / apache/datafusion

from_unixtime does not work for large values

Open
#16,594 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

It looks `from_unixtime` supports only -8334601211038 <= sec <= 8210266876799

### To Reproduce

```
$datafusion-cli
DataFusion CLI v48.0.0
> select from_unixtime(8210266876799, 'America/New_York');
+--------------------------------------------------------------+
| from_unixtime(Int64(8210266876799),Utf8("America/New_York")) |
+--------------------------------------------------------------+
| +262142-12-31T18:59:59-05:00 |
+--------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.039 seconds.

> select from_unixtime(8210266876799 + 1, 'America/New_York');
+----------------------------------------------------------------------------------------------------------------+
| from_unixtime(Int64(8210266876799) + Int64(1),Utf8("America/New_York")) |
+----------------------------------------------------------------------------------------------------------------+
| ERROR: Cast error: Failed to convert 8210266876800 to datetime for Timestamp(Second, Some("America/New_York")) |
+----------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.006 seconds.

> select from_unixtime(-8334601211038, 'America/New_York');
+---------------------------------------------------------------+
| from_unixtime(Int64(-8334601211038),Utf8("America/New_York")) |
+---------------------------------------------------------------+
| -262143-01-01T00:00:00-04:56 |
+---------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.

> select from_unixtime(-8334601211038 -1, 'America/New_York');

thread 'main' panicked at ...
Local time out of range for `NaiveDateTime`
```

### Expected behavior

We should be able to convert all `Int64` values

### Additional context

Spark can convert those large numbers

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two boundary cases in datafusion-cli and trace the from_unixtime conversion for timezone-aware timestamps. Done means all Int64 values convert without a cast error or panic, including the values just beyond the reported limits.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.