apache / apache/arrow

[Python] Converting date32/64 to pandas using nanoseconds can silently overflow

Open
#36,084 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

If you specify to convert a date32 or date64 field to numpy/pandas datetime64 (i.e. not datetime.date objects) using `date_as_object=False`, and your date is out of bounds for the target resolution (at the moment nanoseconds, but with https://github.com/apache/arrow/pull/35656 and recent pandas versions, this will become milliseconds), you silently get mangled values:

```
>>> pa.array([datetime.date(2400, 1, 1)]).to_pandas(date_as_object=False)
0 1815-06-13 00:25:26.290448384
dtype: datetime64[ns]
```

This is because we currently simply multiple the values to get nanoseconds, without bounds / overflow checking:

https://github.com/apache/arrow/blob/b4ac585ecb4da610cc64e346e564ca86594aec53/python/pyarrow/src/arrow/python/arrow_to_pandas.cc#L1592-L1594

We could maybe use a cast instead? (which already has proper bounds checking):

```
>>> pa.array([datetime.date(2400, 1, 1)]).cast(pa.timestamp("ns"))
...
ArrowInvalid: Casting from date32[day] to timestamp[ns] would result in out of bounds timestamp: 157054
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the date32/date64 conversion in Python with date_as_object=False, then inspect arrow/python/src/arrow/python/arrow_to_pandas.cc around the linked conversion code. Compare its behavior with the bounds-checked timestamp cast shown in the issue. Done means out-of-bounds dates are no longer silently mangled, while valid conversions continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.