Reading back (some) Parquet int96 timestamps provides wrong round-trip results
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
PyArrow version: 13.0.0
Python version: 3.10.9
OS: MacOSX, arm64
Some Timestamp values cannot be successfully round-tripped through PyArrow-Parquet[INT96 types]-PyArrow, even though they are within the expressible range of the timeunit ("milliseconds" in the example below).
---
To reproduce:
```
In [1]: import datetime
In [2]: import pyarrow as pa
In [3]: import pyarrow.parquet as papq
In [4]: ts = pa.array([datetime.datetime(1200, 1, 1, 0, 0, 59), datetime.datetime(2000, 1, 1), datetime.datetime(2400, 1, 1)], pa.timestamp("ms"))
In [5]: table = pa.table({"ts": ts})
In [6]: papq.write_table(table, "test.pq", use_deprecated_int96_timestamps=True)
In [7]: papq.read_table("test.pq", coerce_int96_timestamp_unit="ms")
Out[7]:
pyarrow.Table
ts: timestamp[ms]
----
ts: [[1784-07-20 23:35:32.709,2000-01-01 00:00:00.000,2400-01-01 00:00:00.000]]
```
Note that the first timestamp is received as `1784-07-20 23:35:32.709` instead of the expected `1200-01-01 00:00:59.000`.
However, if we change the timestamp from `datetime.datetime(1200, 1, 1, 0, 0, 59)` to `datetime.datetime(1200, 1, 1, 0, 0, 00)` it now seems to work.
```
In [9]: ts = pa.array([datetime.datetime(1200, 1, 1, 0, 0, 00), datetime.datetime(2000, 1, 1), datetime.datetime(2400, 1, 1)], pa.timestamp("ms"))
In [10]: table = pa.table({"ts": ts})
In [11]: papq.write_table(table, "test.pq", use_deprecated_int96_timestamps=True)
In [12]: papq.read_table("test.pq", coerce_int96_timestamp_unit="ms")
Out[12]:
pyarrow.Table
ts: timestamp[ms]
----
ts: [[1200-01-01 00:00:00.000,2000-01-01 00:00:00.000,2400-01-01 00:00:00.000]]
```
### Component(s)
C++, Parquet,Python
Contributor guide
Assessment
This issue has not been assessed yet.