Cast error on roundtrip of categorical column to parquet and back
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Writing a table to parquet, then reading it back fails if:
1. One of the columns is a dictionary (came from a pandas Categorical), **and**
1. Passing the table's schema to `read_table`
Failing on attempt to cast int64 into dictionary (full stack trace below).
This seems related to ARROW-11157 - but even if losing the categorical type when reading from parquet, the reader should not barf when reading with the schema.
Minimal example of failing code:
```java
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
a = [1,2,3,4,1,2,3,4,1,2,3,4]
b = ["a" for i in a]
c = [i for i in range(len(a))]
df = pd.DataFrame({"a":a, "b":b, "c":c})
df['a'] = df['a'].astype('category')
print("df dtypes:\n", df.dtypes)
t = pa.Table.from_pandas(df, preserve_index=True)
s = t.schema
ds.write_dataset(t, format='parquet', base_dir='./test')
df2 = pq.read_table('./test', schema=s).to_pandas()
print("df2 dtypes:\n", df2.dtypes)
```
Which gives:
```java
df dtypes:
a category
b object
c int64
dtype: object
Traceback (most recent call last):
File "/Users/yishai/lab/pyarrow_bug/reproduce.py", line 20, in
df2 = pq.read_table('./test', schema=s).to_pandas()
File "/Users/yishai/lab/pyarrow_bug/venv/lib/python3.9/site-packages/pyarrow/parquet/_init_.py", line 2827, in read_table
return dataset.read(columns=columns, use_threads=use_threads,
File "/Users/yishai/lab/pyarrow_bug/venv/lib/python3.9/site-packages/pyarrow/parquet/_init_.py", line 2473, in read
table = self._dataset.to_table(
File "pyarrow/_dataset.pyx", line 331, in pyarrow._dataset.Dataset.to_table
File "pyarrow/_dataset.pyx", line 2577, in pyarrow._dataset.Scanner.to_table
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from int64 to dictionary using function cast_dictionary
```
**Reporter**: [Yishai Beeri](https://issues.apache.org/jira/browse/ARROW-17625)
**Note**: *This issue was originally created as [ARROW-17625](https://issues.apache.org/jira/browse/ARROW-17625). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by running the minimal Python example using pandas, pyarrow, parquet.read_table, and dataset.write_dataset, then trace the read path that applies the supplied table schema to the dictionary column. Done means the parquet roundtrip completes without the unsupported int64-to-dictionary cast error, even if the categorical type is not preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100