apache / apache/arrow

Reading back (some) Parquet int96 timestamps provides wrong round-trip results

Open
#37,580 2 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

PyArrow version: 13.0.0
Python version: 3.10.9
OS: MacOSX, arm64

Some Timestamp values cannot be successfully round-tripped through PyArrow-Parquet[INT96 types]-PyArrow, even though they are within the expressible range of the timeunit ("milliseconds" in the example below).

---

To reproduce:

```
In [1]: import datetime

In [2]: import pyarrow as pa

In [3]: import pyarrow.parquet as papq

In [4]: ts = pa.array([datetime.datetime(1200, 1, 1, 0, 0, 59), datetime.datetime(2000, 1, 1), datetime.datetime(2400, 1, 1)], pa.timestamp("ms"))

In [5]: table = pa.table({"ts": ts})

In [6]: papq.write_table(table, "test.pq", use_deprecated_int96_timestamps=True)

In [7]: papq.read_table("test.pq", coerce_int96_timestamp_unit="ms")
Out[7]:
pyarrow.Table
ts: timestamp[ms]
----
ts: [[1784-07-20 23:35:32.709,2000-01-01 00:00:00.000,2400-01-01 00:00:00.000]]
```

Note that the first timestamp is received as `1784-07-20 23:35:32.709` instead of the expected `1200-01-01 00:00:59.000`.

However, if we change the timestamp from `datetime.datetime(1200, 1, 1, 0, 0, 59)` to `datetime.datetime(1200, 1, 1, 0, 0, 00)` it now seems to work.

```
In [9]: ts = pa.array([datetime.datetime(1200, 1, 1, 0, 0, 00), datetime.datetime(2000, 1, 1), datetime.datetime(2400, 1, 1)], pa.timestamp("ms"))

In [10]: table = pa.table({"ts": ts})

In [11]: papq.write_table(table, "test.pq", use_deprecated_int96_timestamps=True)

In [12]: papq.read_table("test.pq", coerce_int96_timestamp_unit="ms")
Out[12]:
pyarrow.Table
ts: timestamp[ms]
----
ts: [[1200-01-01 00:00:00.000,2000-01-01 00:00:00.000,2400-01-01 00:00:00.000]]
```

### Component(s)

C++, Parquet,Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.