apache / apache/arrow

pyarrow/pandas interop inconsistent when constructing empty dataframes with non-empty range indices

Open
#40,743 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

```
pa.__version__ => 15.0.2
pandas.__version__ => 3.0.0.dev0+584.g41383cf140
```

Consider:

```python
import pandas as pd
import pyarrow as pa
from pandas.testing import assert_frame_equal
from io import BytesIO

data = pd.DataFrame(index=pd.RangeIndex(0, 10), data=None)

tab = pa.table(data)

buf = BytesIO()

pa.parquet.write_table(tab, buf)

got = pa.parquet.read_table(buf)

assert tab.shape == got.shape # False

# and hence
expect = data

got_tab = tab.to_pandas()

assert_frame_equal(expect, got_tab) # True

got_pq = got.to_pandas()
assert_frame_equal(expect, got_pq) # False

# and hence
buf = BytesIO()
data.to_parquet(buf, index=None)
got = pd.read_parquet(buf)

assert_frame_equal(data, got) # False
```

The round-trip through `pa.table` was fixed in #26599.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.