apache / apache/arrow

[python] `dtype` of `Categorical` category columns are not preserved

Open
#33,070 0 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Hi there,

First time submitting an issue here so apologies if there's anything I've missed.

I see the below bug, where by the `dtype` of the categories themselves (within a `pd.Categorical` are not preserved on a round trip via pyarrow. Hopefully the snippet below demonstrates the issue.

The reason this causes an issue, is because the dtypes need to be the same in order for the categories to be considered the same (so they can then be concatenated, for example).

Current workaround is to store as a plain `pd.StringDtype()` and then convert to `pd.Categorical` in memory with Pandas (which infers from the underlying type, but in doing so sacrifices disk saving of storing as a dictionary).

Using pyarrow 9.0.0 and pandas 1.4.4.

Thanks
 

`import pandas as pd`

`import pyarrow as pa`

 

`{}# note, Categorical column B is constructed from `pd.{`}`{}StringDtype`{`}

`df = pd.DataFrame(\{"A": ["a", "b", "c", "a"]\}, dtype=pd.StringDtype())`

`df["B"] = df["A"].astype("category")`

`print(df["B"].cat.categories)`
`# Index(['a', 'b', 'c'], dtype='string')`

 

`# however, this is downcast to `object` during a roundtrip`

`print(pa.Table.from_pandas(df).to_pandas()["B"].cat.categories)`

`# Index(['a', 'b', 'c'], dtype='object')`

 

 

**Reporter**: [Ryan Ballard](https://issues.apache.org/jira/browse/ARROW-17852)

**Note**: *This issue was originally created as [ARROW-17852](https://issues.apache.org/jira/browse/ARROW-17852). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the reported pandas and pyarrow round-trip with pa.Table.from_pandas(df).to_pandas() and inspect the categorical column's categories. Trace the Python conversion entry points involved; done means the category values retain their original pandas StringDtype after the round trip, while existing behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.