apache / apache/arrow

[Python] pyarrow fails to write and read a dataframe with MultiIndex containing a RangeIndex with Pandas 1.5.0

Open
#33,030 3 comments 0 reactions 1 assignee Claimed by @GianlucaFicarelli View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

A dataframe with a MultiIndex built in this way:
```java

import pandas as pd
df1 = pd.DataFrame({"a": [10, 11, 12], "b": [20, 21, 22]}, index=pd.RangeIndex(3, name="idx0"))
df1 = df1.set_index("b", append=True)
print(df1)
print(df1.index.get_level_values("idx0"))
```
gives with Pandas 1.5.0:
```java

          a
idx0 b     
0    20  10
1    21  11
2    22  12

RangeIndex(start=0, stop=3, step=1, name='idx0')
```
while with Pandas 1.4.4:
```java

        a
idx0 b     
0    20  10
1    21  11
2    22  12

Int64Index([0, 1, 2], dtype='int64', name='idx0')
```
i.e. the result is RangeIndex instead of Int64Index.

With pandas 1.5.0 and pyarrow 9.0.0, writing this DataFrame with index=None (i.e. the default value) as in:
```java

df1.to_parquet(path, engine="pyarrow", index=None)
```
then reading the same file with:
```java

pd.read_parquet(path, engine="pyarrow")
```
raises an exception:
```java

File //lib/python3.9/site-packages/pyarrow/pandas_compat.py:997, in _extract_index_level(table, result_table, field_name, field_name_to_metadata)
    995 def _extract_index_level(table, result_table, field_name,
    996                          field_name_to_metadata):
--> 997     logical_name = field_name_to_metadata[field_name]['name']
    998     index_name = _backwards_compatible_index_name(field_name, logical_name)
    999     i = table.schema.get_field_index(field_name)

KeyError: 'b'
```
while with pandas 1.4.4 and pyarrow 9.0.0 it works correctly. 

Note that the problem disappears if the parquet file is written with index=True (that is not the default value), probably because the RangeIndex is converted to Int64Index:
```java

df1.to_parquet(path, engine="pyarrow", index=True)
```
I suspect that the issue is caused by the change from Int64Index to RangeIndex and it may be related to

Should pyarrow be able to handle this case? Or is it an issue with Pandas?

**Reporter**: [Gianluca Ficarelli](https://issues.apache.org/jira/browse/ARROW-17806)

**Note**: *This issue was originally created as [ARROW-17806](https://issues.apache.org/jira/browse/ARROW-17806). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.