apache / apache/arrow

`pa.Table.to_pandas(strings_to_categorical=True)` failing for large string/binary arrays

Open
#44,340 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I have a table with lots of strings that I would like to export to Pandas. The following code can recreate the error:

```python
import numpy as np
import pyarrow as pa

SIZE = 1024
N = 2 * 1024 * 1024

buffer = np.random.bytes(N * SIZE)
table = pa.Table.from_pydict({
"row": [buffer[i * SIZE: (i + 1) * SIZE] for i in range(N)]
})
df = table.to_pandas(strings_to_categorical=True)
```

This is currently failing with the error:
```
Traceback (most recent call last):
File "/home/alandu/workspace/scratch/repro.py", line 13, in
df = table.to_pandas(strings_to_categorical=True)
File "pyarrow/array.pxi", line 885, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 5002, in pyarrow.lib.Table._to_pandas
File "/home/alandu/micromamba/envs/test/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 784, in table_to_dataframe
result = pa.lib.table_to_blocks(options, table, categories,
File "pyarrow/table.pxi", line 3941, in pyarrow.lib.table_to_blocks
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```

This is using Python 3.10 on PyArrow 17.0 on Linux (installed via conda-forge).

This *only* seems to happen when I set `strings_to_categorical=True` -- if that is `False`, then I can export this to a dataframe without issues.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.