[Python] Categorical boolean column saved as regular boolean in parquet
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
When saving a pandas dataframe to parquet, if there is a categorical column where the categories are boolean, the column is saved as regular boolean.
This causes an issue because, when reading back the parquet file, I expect the column to still be categorical.
Reproducible example:
```python
import pandas as pd
import pyarrow
# Create dataframe with boolean column that is then converted to categorical
df = pd.DataFrame({'a': [True, True, False, True, False]})
df['a'] = df['a'].astype('category')
# Convert to arrow Table and save to disk
table = pyarrow.Table.from_pandas(df)
pyarrow.parquet.write_table(table, 'test.parquet')
# Reload data and convert back to pandas
table_rel = pyarrow.parquet.read_table('test.parquet')
df_rel = table_rel.to_pandas()
```
The arrow `table` variable correctly converts the column to an arrow `DICTIONARY` type:
```
>>> df['a']
0 True
1 True
2 False
3 True
4 False
Name: a, dtype: category
Categories (2, object): [False, True]
>>>
>>> table
pyarrow.Table
a: dictionary
```
However, the reloaded column is now a regular boolean:
```
>>> table_rel
pyarrow.Table
a: bool
>>>
>>> df_rel['a']
0 True
1 True
2 False
3 True
4 False
Name: a, dtype: bool
```
I would have expected the column to be read back as categorical.
**Reporter**: [Joao Moreira](https://issues.apache.org/jira/browse/ARROW-13342)
#### Related issues:
- [[C++][Parquet] Support direct dictionary decoding of types other than BYTE_ARRAY](https://github.com/apache/arrow/issues/22534) (is blocked by)
- [[Python] Consistent handling of categoricals](https://github.com/apache/arrow/issues/27067) (is related to)
**Note**: *This issue was originally created as [ARROW-13342](https://issues.apache.org/jira/browse/ARROW-13342). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the Python reproduction using pyarrow.Table.from_pandas, parquet.write_table, and parquet.read_table, then inspect how dictionary-encoded boolean columns are converted on read. Done means the reproduced categorical column remains categorical after the parquet round trip, with coverage for this case; related dictionary-decoding work may be relevant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100