[Python] Cannot create pandas categorical from table only with nulls
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
A pyarrow Table with only null values cannot be instantiated as a Pandas DataFrame with said column as a category. However, pandas does support "empty" categoricals. Therefore, a simple patch would be to load the pa.Table as an object first and convert, once in pandas, to a categorical which will be empty. However, that does not solve the pyarrow bug at its root.
Sample reproducible example
```java
import pyarrow as pa
pylist = [{'x': None, '__index_level_0__': 2}, {'x': None, '__index_level_0__': 3}]
tbl = pa.Table.from_pylist(pylist)
# Errors
df_broken = tbl.to_pandas(categories=["x"])
# Works
df_works = tbl.to_pandas()
df_works = df_works.astype({"x": "category"})
```
**Environment**: OSX 12.6
M1 silicon
**Reporter**: [Damian Barabonkov](https://issues.apache.org/jira/browse/ARROW-18099) / @DamianBarabonkovQC
**Assignee**: [Damian Barabonkov](https://issues.apache.org/jira/browse/ARROW-18099) / @DamianBarabonkovQC
#### PRs and other links:
- [GitHub Pull Request #14953](https://github.com/apache/arrow/pull/14953)
**Note**: *This issue was originally created as [ARROW-18099](https://issues.apache.org/jira/browse/ARROW-18099). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by running the provided Python reproduction with pyarrow and pandas, then inspect pull request #14953 for the existing investigation. Confirm the failure occurs when converting an all-null table column requested as a category, and consider the issue complete when tbl.to_pandas(categories=["x"]) succeeds with the expected empty categorical result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100