[Python] Pyarrow table conversion from pandas fails for categorical fields with arrow dtypes
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the enhancement requested
There doesn't seem to be a way to convert a pandas column using the new pyarrow backend to a DictionaryArray in arrow. Both categorical and non-categorical columns of ArrowDtypes fail conversion to Table with different errors.
Versions:
* pandas 2.0.0
* pyarrow 11.0.0
Platform: OSX
```python
import pyarrow as pa
import pandas as pd
schema = pa.schema([
pa.field(
name='category',
type=pa.dictionary(index_type=pa.int8(), value_type=pa.string())
),
])
df = pd.DataFrame({'category': ['a']})
# works
pa.Table.from_pandas(df, schema=schema)
# works
df2 = df.assign(category=df['category'].astype('category'))
pa.Table.from_pandas(df2, schema=schema)
# fails
df3 = df.assign(category=df['category'].astype(pd.ArrowDtype(pa.string())))
pa.Table.from_pandas(df3, schema=schema)
# fails
df4 = df.assign(category=df['category'].astype(pd.ArrowDtype(pa.string())).astype('category'))
pa.Table.from_pandas(df4, schema=schema)
```
df3 fails with
```
pa.Table.from_pandas(df3, schema=schema)
File "pyarrow/table.pxi", line 3572, in pyarrow.lib.Table.from_pandas
File "pyarrow/table.pxi", line 3657, in pyarrow.lib.Table.from_arrays
File "pyarrow/table.pxi", line 1421, in pyarrow.lib._sanitize_arrays
File "pyarrow/array.pxi", line 347, in pyarrow.lib.asarray
File "pyarrow/table.pxi", line 523, in pyarrow.lib.ChunkedArray.cast
File "/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/compute.py", line 391, in cast
return call_function("cast", [arr], options)
File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from string to dictionary using function cast_dictionary
```
df4 fails with
```
pa.Table.from_pandas(df4, schema=schema)
File "pyarrow/table.pxi", line 3557, in pyarrow.lib.Table.from_pandas
File "/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, in dataframe_to_arrays
arrays = [convert_column(c, f)
File "/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, in
arrays = [convert_column(c, f)
File "/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, in convert_column
raise e
File "/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, in convert_column
result = pa.array(col, type=type_, from_pandas=True, safe=safe)
File "pyarrow/array.pxi", line 310, in pyarrow.lib.array
File "pyarrow/array.pxi", line 2608, in pyarrow.lib.DictionaryArray.from_arrays
File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: ("Could not convert with type pyarrow.lib.StringScalar: did not recognize Python value type when inferring an Arrow data type", 'Conversion failed for column category with type category')
```
### Component(s)
Python
Contributor guide
Research direction
Start at pa.Table.from_pandas and follow the conversion path shown through pyarrow/pandas_compat.py, especially dataframe_to_arrays and convert_column. Reproduce the df3 and df4 cases with the stated pandas and pyarrow versions, then verify that ArrowDtype columns convert to the requested dictionary schema without either reported error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100