apache / apache/arrow

[Python] Categorical boolean column saved as regular boolean in parquet

Open
#29,017 7 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

When saving a pandas dataframe to parquet, if there is a categorical column where the categories are boolean, the column is saved as regular boolean.

This causes an issue because, when reading back the parquet file, I expect the column to still be categorical.

 
Reproducible example:
```python

import pandas as pd
import pyarrow

# Create dataframe with boolean column that is then converted to categorical
df = pd.DataFrame({'a': [True, True, False, True, False]})
df['a'] = df['a'].astype('category')

# Convert to arrow Table and save to disk
table = pyarrow.Table.from_pandas(df)
pyarrow.parquet.write_table(table, 'test.parquet')

# Reload data and convert back to pandas
table_rel = pyarrow.parquet.read_table('test.parquet')
df_rel = table_rel.to_pandas()
```

The arrow `table` variable correctly converts the column to an arrow `DICTIONARY` type:
```

>>> df['a']
0 True
1 True
2 False
3 True
4 False
Name: a, dtype: category
Categories (2, object): [False, True]
>>>
>>> table
pyarrow.Table
a: dictionary
```

However, the reloaded column is now a regular boolean:
```

>>> table_rel
pyarrow.Table
a: bool
>>>
>>> df_rel['a']
0 True
1 True
2 False
3 True
4 False
Name: a, dtype: bool
```

I would have expected the column to be read back as categorical.

**Reporter**: [Joao Moreira](https://issues.apache.org/jira/browse/ARROW-13342)
#### Related issues:
- [[C++][Parquet] Support direct dictionary decoding of types other than BYTE_ARRAY](https://github.com/apache/arrow/issues/22534) (is blocked by)
- [[Python] Consistent handling of categoricals](https://github.com/apache/arrow/issues/27067) (is related to)

**Note**: *This issue was originally created as [ARROW-13342](https://issues.apache.org/jira/browse/ARROW-13342). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python reproduction using pyarrow.Table.from_pandas, parquet.write_table, and parquet.read_table, then inspect how dictionary-encoded boolean columns are converted on read. Done means the reproduced categorical column remains categorical after the parquet round trip, with coverage for this case; related dictionary-decoding work may be relevant.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.