[python] `dtype` of `Categorical` category columns are not preserved
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Hi there,
First time submitting an issue here so apologies if there's anything I've missed.
I see the below bug, where by the `dtype` of the categories themselves (within a `pd.Categorical` are not preserved on a round trip via pyarrow. Hopefully the snippet below demonstrates the issue.
The reason this causes an issue, is because the dtypes need to be the same in order for the categories to be considered the same (so they can then be concatenated, for example).
Current workaround is to store as a plain `pd.StringDtype()` and then convert to `pd.Categorical` in memory with Pandas (which infers from the underlying type, but in doing so sacrifices disk saving of storing as a dictionary).
Using pyarrow 9.0.0 and pandas 1.4.4.
Thanks
`import pandas as pd`
`import pyarrow as pa`
`{}# note, Categorical column B is constructed from `pd.{`}`{}StringDtype`{`}
`df = pd.DataFrame(\{"A": ["a", "b", "c", "a"]\}, dtype=pd.StringDtype())`
`df["B"] = df["A"].astype("category")`
`print(df["B"].cat.categories)`
`# Index(['a', 'b', 'c'], dtype='string')`
`# however, this is downcast to `object` during a roundtrip`
`print(pa.Table.from_pandas(df).to_pandas()["B"].cat.categories)`
`# Index(['a', 'b', 'c'], dtype='object')`
**Reporter**: [Ryan Ballard](https://issues.apache.org/jira/browse/ARROW-17852)
**Note**: *This issue was originally created as [ARROW-17852](https://issues.apache.org/jira/browse/ARROW-17852). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by running the reported pandas and pyarrow round-trip with pa.Table.from_pandas(df).to_pandas() and inspect the categorical column's categories. Trace the Python conversion entry points involved; done means the category values retain their original pandas StringDtype after the round trip, while existing behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100