apache / apache/arrow

[Python] Timestamp - out of bounds for nanoseconds

Open
#44,420 0 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

**Environment**
OS: Windows/Linux
Python: 3.11.2
Pyarrow: 17.0.0
Pandas: 2.2.2

**Description**
When trying to read a timestamp value, below the [pandas min. value of 1677-09-21 00:12:43.145224193](https://pandas.pydata.org/docs/reference/api/pandas.Timestamp.min.html), from a datetime object into a pyarrow table, the result is an out of bounds for nanoseconds exception.

I have found problems that might relate but did not solve the issue here

- [ARROW-5359](https://issues.apache.org/jira/browse/ARROW-5359)
- [ARROW-3448](https://issues.apache.org/jira/browse/ARROW-3448)

**Example Code**
```python
import pyarrow as pa
import datetime

schema = pa.schema([])
schema = schema.append(pa.field("CreateAt", pa.timestamp(unit="ns")))

ts = datetime.datetime(1677, 9, 21, 1) # OK
arrays = [[ts]]
print(arrays)
table = pa.Table.from_arrays(arrays, schema=schema)
print(table)

ts = datetime.datetime(1, 1, 1, 1) # NoK
arrays = [[ts]]
print(arrays)
table = pa.Table.from_arrays(arrays, schema=schema)
print(table)
```

**Use Case**
I am reading data from a database, where one column has ns precision timestamps. Instead of null values, it uses `0001-01-01 00:00:00.0000000`. The goal is to store the result of the database read, which is an array containing datetime objects, into a Pyarrow table to then store it as parquet. This works well, until i hit a timestamp too big or small for pandas.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.