apache / apache/arrow

[Python][Parquet] `timestamp[s]` does not round-trip `parquet` serialization.

Open
#41,382 5 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Timestamps with second resolution get upcasted to millisecond resolution when serializing and deserializing. They should either round trip, or there should be a warning/error when attempting to serialize them.

```python
from datetime import datetime
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow import parquet

dates = [
datetime(2021, 1, 1, 0, 0, 3),
datetime(2021, 1, 1, 0, 0, 4),
datetime(2021, 1, 1, 0, 0, 5),
]

table = pa.table({"time": pa.array(dates, type=pa.timestamp("s"))})
print(table.schema) # timestamp[s]
parquet.write_table(table, "timestamp_roundtrip.parquet")
table2 = parquet.read_table("timestamp_roundtrip.parquet")
print(table2.schema) # timestamp[ms]
```

Tested with pyarrow 16.0.0

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.