apache / apache/arrow

Cast error on roundtrip of categorical column to parquet and back

Open
#32,869 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Writing a table to parquet, then reading it back fails if:
1. One of the columns is a dictionary (came from a pandas Categorical), **and**
1. Passing the table's schema to `read_table`

Failing on attempt to cast int64 into dictionary (full stack trace below).

This seems related to ARROW-11157 - but even if losing the categorical type when reading from parquet, the reader should not barf when reading with the schema.

Minimal example of failing code:
```java

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
a = [1,2,3,4,1,2,3,4,1,2,3,4]
b = ["a" for i in a]
c = [i for i in range(len(a))]
df = pd.DataFrame({"a":a, "b":b, "c":c})
df['a'] = df['a'].astype('category')
print("df dtypes:\n", df.dtypes)
t = pa.Table.from_pandas(df, preserve_index=True)
s = t.schema
ds.write_dataset(t, format='parquet', base_dir='./test')
df2 = pq.read_table('./test', schema=s).to_pandas()
print("df2 dtypes:\n", df2.dtypes)
```
 

Which gives: 
```java

df dtypes:
a    category
b      object
c       int64
dtype: object
Traceback (most recent call last):
  File "/Users/yishai/lab/pyarrow_bug/reproduce.py", line 20, in
  df2 = pq.read_table('./test', schema=s).to_pandas()
  File "/Users/yishai/lab/pyarrow_bug/venv/lib/python3.9/site-packages/pyarrow/parquet/_init_.py", line 2827, in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
  File "/Users/yishai/lab/pyarrow_bug/venv/lib/python3.9/site-packages/pyarrow/parquet/_init_.py", line 2473, in read
    table = self._dataset.to_table(
  File "pyarrow/_dataset.pyx", line 331, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 2577, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from int64 to dictionary using function cast_dictionary
```

**Reporter**: [Yishai Beeri](https://issues.apache.org/jira/browse/ARROW-17625)

**Note**: *This issue was originally created as [ARROW-17625](https://issues.apache.org/jira/browse/ARROW-17625). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal Python example using pandas, pyarrow, parquet.read_table, and dataset.write_dataset, then trace the read path that applies the supplied table schema to the dictionary column. Done means the parquet roundtrip completes without the unsupported int64-to-dictionary cast error, even if the categorical type is not preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.