apache / apache/arrow

[Python] Cannot create pandas categorical from table only with nulls

Open
#33,297 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

A pyarrow Table with only null values cannot be instantiated as a Pandas DataFrame with said column as a category. However, pandas does support "empty" categoricals. Therefore, a simple patch would be to load the pa.Table as an object first and convert, once in pandas, to a categorical which will be empty. However, that does not solve the pyarrow bug at its root.

 

Sample reproducible example
```java

import pyarrow as pa
pylist = [{'x': None, '__index_level_0__': 2}, {'x': None, '__index_level_0__': 3}]
tbl = pa.Table.from_pylist(pylist)
 

# Errors

df_broken = tbl.to_pandas(categories=["x"])
 

# Works
df_works = tbl.to_pandas()
df_works = df_works.astype({"x": "category"})
```

**Environment**: OSX 12.6
M1 silicon
**Reporter**: [Damian Barabonkov](https://issues.apache.org/jira/browse/ARROW-18099) / @DamianBarabonkovQC
**Assignee**: [Damian Barabonkov](https://issues.apache.org/jira/browse/ARROW-18099) / @DamianBarabonkovQC
#### PRs and other links:
- [GitHub Pull Request #14953](https://github.com/apache/arrow/pull/14953)

**Note**: *This issue was originally created as [ARROW-18099](https://issues.apache.org/jira/browse/ARROW-18099). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the provided Python reproduction with pyarrow and pandas, then inspect pull request #14953 for the existing investigation. Confirm the failure occurs when converting an all-null table column requested as a category, and consider the issue complete when tbl.to_pandas(categories=["x"]) succeeds with the expected empty categorical result.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.