apache / apache/arrow

[Python/C++] Empty column selection no longer restores index

Open
#21,621 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The index of a dataframe is no longer reconstructed when using empty column selection. This is a regression to 0.12.1 and probably only happens for pd.RangeIndex

```python

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from kartothek.serialization import ParquetSerializer
from storefact import get_store_from_url
print(pa.__version__)
df = pd.DataFrame(
{"a": [1, 2]}
)
print(df.index)

table = pa.Table.from_pandas(df)
buf = pa.BufferOutputStream()
pq.write_table(table, buf)
reader = pa.BufferReader(buf.getvalue().to_pybytes())
table_restored = pq.read_pandas(reader, columns=[])
df_restored = table_restored.to_pandas()
print(len(df_restored))
```

**Reporter**: [Florian Jetter](https://issues.apache.org/jira/browse/ARROW-5139) / @fjetter
#### Related issues:
- [[Python] RangeIndex serialization change implications](https://github.com/apache/arrow/issues/21880) (is related to)

**Note**: *This issue was originally created as [ARROW-5139](https://issues.apache.org/jira/browse/ARROW-5139). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Python reproduction and inspect the path triggered by pq.read_pandas(reader, columns=[]), focusing on how RangeIndex metadata is restored after empty-column selection. Done means the restored dataframe retains the original RangeIndex and row count without affecting ordinary column reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.