[Python] Table.to_pandas() fails for dictionary encoded columns with an is_null partition_expression
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Minimal steps to reproduce:
I attached a `.zip` file containing a python script and a test parquet file. Running this python script reproduces the issue.
The steps taken to reproduce:
1. Create a test parquet file with one column containing only null.
1. Create a parquet fragment from this file adding a `partition_expression` with an `is_null` guarantee on this fragment.
1. Create a `FileSystemDataset` from this fragment setting the schema to be a dictionary column.
1. Call `.to_table().to_pandas()` on the resulting pyarrow dataset. You will get the following error.
```java
File "/.../pip-core_pandas/pandas/core/dtypes/dtypes.py", line 492, in validate_categories
raise ValueError("Categorical categories cannot be null")
ValueError: Categorical categories cannot be null
```
My understanding of why this doesn't work:
1. There are 2 ways of dictionary encoding nulls: `mask` and `encode` described in the [pyarrow docs](https://arrow.apache.org/docs/python/generated/pyarrow.compute.DictionaryEncodeOptions.html#pyarrow.compute.DictionaryEncodeOptions). Pyarrow supports both but pandas categoricals only supports mask. Arguably the real issue here is pandas should support `encode` style categoricals.
1. When you provide an `.is_null` guarantee on a fragment arrow will not actually read the data. It knows the type from the schema, we've guaranteed the values are all null and it can get the length from the parquet metadata so it has everything it needs.
1. Instead of reading the data it uses the [Null ArrayFactory](https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/util.cc). For dictionary type columns I believe that calls [this DictionaryArray constructor ](https://github.com/apache/arrow/blob/53752adc6b81166cd4ee7db5a819494042f29197/cpp/src/arrow/array/array_dict.cc#L80-L93)which appears to be creating the dictionary in the `encode` style.
Would it be possible to make this configurable? It seems like the `mask` style of dictionary encoding is the default for the rest of PyArrow and it would solve the Pandas compatibility issue. I appreciate this is probably an extremely niche issue but my options for a workaround are looking pretty horrible.
**Environment**: Ubuntu 18.04, PyArrow 8.0.0, Pandas 1.4.3
**Reporter**: [Thomas Newton](https://issues.apache.org/jira/browse/ARROW-16905)
#### Original Issue Attachments:
- [reproduce_null_dictionary_issue.zip](https://issues.apache.org/jira/secure/attachment/13045627/reproduce_null_dictionary_issue.zip)
**Note**: *This issue was originally created as [ARROW-16905](https://issues.apache.org/jira/browse/ARROW-16905). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by running the attached reproduction script and inspect the failing Table.to_table().to_pandas() path. Read cpp/src/arrow/array/util.cc and the referenced DictionaryArray constructor in cpp/src/arrow/array/array_dict.cc, then verify that the reproduction converts successfully without pandas rejecting null categorical categories.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100