empty table changes datatpye from ns to us
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Hello!
I don't know if this is specific to python but seems quite straightforward. I didn't look into it very deeply but its easy to reproduce so I thought I'd post it.
```python
import pyarrow as pa
import pyarrow.parquet as pq
pa.__version__
```
> '11.0.0'
```python
pt = pa.Table.from_pylist([], pa.schema([("d", pa.timestamp("ns"))]))
pt
```
```
pyarrow.Table
d: timestamp[ns]
----
d: [[]]
```
```python
pq.write_table(pt, "_.p")
pq.read_table("_.p")
```
```
pyarrow.Table
d: timestamp[us]
----
d: [[]]
```
very interestingly if I use pandas, it goes the other way around
```python
import pandas as pd
pt = pa.Table.from_pandas(pd.DataFrame({"d": pd.date_range("2023-01-01", "2023-01-02")}))
pt
```
```
pyarrow.Table
d: timestamp[ns]
----
d: [[2023-01-01 00:00:00.000000000,2023-01-02 00:00:00.000000000]]
```
```python
pq.write_table(pt, "_.p")
pq.read_table("_.p")
```
```
pyarrow.Table
d: timestamp[us]
----
d: [[2023-01-01 00:00:00.000000,2023-01-02 00:00:00.000000]]
```
### Component(s)
Python
Contributor guide
Research direction
Reproduce the timestamp round trip using the Python entry points pq.write_table and pq.read_table with an empty pyarrow table and a timestamp[ns] schema. Trace the Parquet write/read path to determine why the unit becomes timestamp[us], then add coverage showing that the timestamp unit is preserved for empty tables; the existing non-empty pandas example provides a comparison.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100