Implement export of list of fixed sized binary to pandas
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
When you attempt to create a pandas dataframe from a pyarrow table that contains a list of fixed sized bytes, it generates the error:
```
ArrowNotImplementedError: Not implemented type for Arrow list to pandas: fixed_size_binary[32]
```
Here is a minimal reproducible example:
```
import pyarrow as pa
import pandas as pd
# Create some sample 32-byte binary data
data = [
b'\x00' * 32, # All zeros
b'\xff' * 32, # All ones
b'\x01\x02\x03\x04' * 8, # Repeating pattern
bytes(range(32)), # Sequential bytes 0-31
b'Hello World!' + b'\x00' * 20, # Text padded with zeros
]
# Create a table with a fixed-size binary column
table = pa.table({
'id': [1, 2, 3, 4, 5],
'fixed_binary': pa.array(data, type=pa.binary(32))
})
# This works fine - when you have only a fixed size binary
print(table.to_pandas())
data_as_array = pa.array(data, type=pa.binary(32))
table2 = pa.table({
"list_fixed_binary": pa.array([data_as_array], type=pa.list_(pa.binary(32)))
})
# This fails - when you have a list of fixed sized binary
table2.to_pandas()
```
The error generated is:
```
---------------------------------------------------------------------------
ArrowNotImplementedError Traceback (most recent call last)
Cell In[17], line 1
----> 1 table2.to_pandas()
File [~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/array.pxi:1020](http://localhost:8888/lab/workspaces/auto-w/tree/~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/array.pxi#line=1019), in pyarrow.lib._PandasConvertible.to_pandas()
File [~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/table.pxi:5177](http://localhost:8888/lab/workspaces/auto-w/tree/~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/table.pxi#line=5176), in pyarrow.lib.Table._to_pandas()
File [~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/pandas_compat.py:806](http://localhost:8888/lab/workspaces/auto-w/tree/~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/pandas_compat.py#line=805), in table_to_dataframe(options, table, categories, ignore_metadata, types_mapper)
803 columns = _deserialize_column_index(table, all_columns, column_indexes)
805 column_names = table.column_names
--> 806 result = pa.lib.table_to_blocks(options, table, categories,
807 list(ext_columns_dtypes.keys()))
808 if _pandas_api.is_ge_v3():
809 from pandas.api.internals import create_dataframe_from_blocks
File [~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/table.pxi:4103](http://localhost:8888/lab/workspaces/auto-w/tree/~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/table.pxi#line=4102), in pyarrow.lib.table_to_blocks()
File [~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/error.pxi:92](http://localhost:8888/lab/workspaces/auto-w/tree/~/working/tmp/.venv/lib/python3.12/site-packages/pyarrow/error.pxi#line=91), in pyarrow.lib.check_status()
ArrowNotImplementedError: Not implemented type for Arrow list to pandas: fixed_size_binary[32]
```
In contrast, recent versions of Pandas *does* support these data types if you use the pycapsule interface for arrow.
```
# This works just fine
print(pd.DataFrame.from_records(table2))
```
Produces output like expected:
```
0
0 (b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0...
```
This was tested on pyarrow 21.0.0 and pandas 2.3.3
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.