apache / apache/arrow

empty table changes datatpye from ns to us

Open
#34,782 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hello!

I don't know if this is specific to python but seems quite straightforward. I didn't look into it very deeply but its easy to reproduce so I thought I'd post it.

```python
import pyarrow as pa
import pyarrow.parquet as pq

pa.__version__
```
> '11.0.0'

```python
pt = pa.Table.from_pylist([], pa.schema([("d", pa.timestamp("ns"))]))

pt
```
```
pyarrow.Table
d: timestamp[ns]
----
d: [[]]
```

```python
pq.write_table(pt, "_.p")

pq.read_table("_.p")
```

```
pyarrow.Table
d: timestamp[us]
----
d: [[]]
```

very interestingly if I use pandas, it goes the other way around

```python
import pandas as pd

pt = pa.Table.from_pandas(pd.DataFrame({"d": pd.date_range("2023-01-01", "2023-01-02")}))

pt
```

```
pyarrow.Table
d: timestamp[ns]
----
d: [[2023-01-01 00:00:00.000000000,2023-01-02 00:00:00.000000000]]
```

```python
pq.write_table(pt, "_.p")

pq.read_table("_.p")
```

```
pyarrow.Table
d: timestamp[us]
----
d: [[2023-01-01 00:00:00.000000,2023-01-02 00:00:00.000000]]
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the timestamp round trip using the Python entry points pq.write_table and pq.read_table with an empty pyarrow table and a timestamp[ns] schema. Trace the Parquet write/read path to determine why the unit becomes timestamp[us], then add coverage showing that the timestamp unit is preserved for empty tables; the existing non-empty pandas example provides a comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.