apache / apache/arrow

[Python] Table.to_pandas() fails for dictionary encoded columns with an is_null partition_expression

Open
#32,225 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Minimal steps to reproduce:

I attached a `.zip` file containing a python script and a test parquet file. Running this python script reproduces the issue.

The steps taken to reproduce:
1. Create a test parquet file with one column containing only null.
1. Create a parquet fragment from this file adding a `partition_expression` with an `is_null` guarantee on this fragment.
1. Create a `FileSystemDataset` from this fragment setting the schema to be a dictionary column.
1. Call `.to_table().to_pandas()` on the resulting pyarrow dataset. You will get the following error.

```java

  File "/.../pip-core_pandas/pandas/core/dtypes/dtypes.py", line 492, in validate_categories
    raise ValueError("Categorical categories cannot be null")
ValueError: Categorical categories cannot be null
```
 

My understanding of why this doesn't work:
1. There are 2 ways of dictionary encoding nulls: `mask` and `encode` described in the [pyarrow docs](https://arrow.apache.org/docs/python/generated/pyarrow.compute.DictionaryEncodeOptions.html#pyarrow.compute.DictionaryEncodeOptions). Pyarrow supports both but pandas categoricals only supports mask. Arguably the real issue here is pandas should support `encode` style categoricals.
1. When you provide an `.is_null` guarantee on a fragment arrow will not actually read the data. It knows the type from the schema, we've guaranteed the values are all null and it can get the length from the parquet metadata so it has everything it needs.
1. Instead of reading the data it uses the [Null ArrayFactory](https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/util.cc). For dictionary type columns I believe that calls [this DictionaryArray constructor ](https://github.com/apache/arrow/blob/53752adc6b81166cd4ee7db5a819494042f29197/cpp/src/arrow/array/array_dict.cc#L80-L93)which appears to be creating the dictionary in the `encode` style.

Would it be possible to make this configurable? It seems like the `mask` style of dictionary encoding is the default for the rest of PyArrow and it would solve the Pandas compatibility issue. I appreciate this is probably an extremely niche issue but my options for a workaround are looking pretty horrible. 

**Environment**: Ubuntu 18.04, PyArrow 8.0.0, Pandas 1.4.3
**Reporter**: [Thomas Newton](https://issues.apache.org/jira/browse/ARROW-16905)
#### Original Issue Attachments:
- [reproduce_null_dictionary_issue.zip](https://issues.apache.org/jira/secure/attachment/13045627/reproduce_null_dictionary_issue.zip)

**Note**: *This issue was originally created as [ARROW-16905](https://issues.apache.org/jira/browse/ARROW-16905). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the attached reproduction script and inspect the failing Table.to_table().to_pandas() path. Read cpp/src/arrow/array/util.cc and the referenced DictionaryArray constructor in cpp/src/arrow/array/array_dict.cc, then verify that the reproduction converts successfully without pandas rejecting null categorical categories.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.