[Python] Converting date32/64 to pandas using nanoseconds can silently overflow
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
If you specify to convert a date32 or date64 field to numpy/pandas datetime64 (i.e. not datetime.date objects) using `date_as_object=False`, and your date is out of bounds for the target resolution (at the moment nanoseconds, but with https://github.com/apache/arrow/pull/35656 and recent pandas versions, this will become milliseconds), you silently get mangled values:
```
>>> pa.array([datetime.date(2400, 1, 1)]).to_pandas(date_as_object=False)
0 1815-06-13 00:25:26.290448384
dtype: datetime64[ns]
```
This is because we currently simply multiple the values to get nanoseconds, without bounds / overflow checking:
https://github.com/apache/arrow/blob/b4ac585ecb4da610cc64e346e564ca86594aec53/python/pyarrow/src/arrow/python/arrow_to_pandas.cc#L1592-L1594
We could maybe use a cast instead? (which already has proper bounds checking):
```
>>> pa.array([datetime.date(2400, 1, 1)]).cast(pa.timestamp("ns"))
...
ArrowInvalid: Casting from date32[day] to timestamp[ns] would result in out of bounds timestamp: 157054
```
### Component(s)
Python
Contributor guide
Research direction
Reproduce the date32/date64 conversion in Python with date_as_object=False, then inspect arrow/python/src/arrow/python/arrow_to_pandas.cc around the linked conversion code. Compare its behavior with the bounds-checked timestamp cast shown in the issue. Done means out-of-bounds dates are no longer silently mangled, while valid conversions continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100