apache / apache/arrow

[Python] pa.DictionaryArray.from_arrays(..) can't convert large numpy / ndarrays

Open
#47,246 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Numpy arrays seem to get converted with pyarrow.array(..) either to pyarrow.Array OR pyarrow.ChunkedArray depending on its' size according to the documentation.

The pa.DictionaryArray.from_arrays(..) is also specified, that it can treat ndarrays, but in case the ndarray is a large object, there is an internal exception which is not able to treat the ChunkedArray object.

```
import pyarrow as pa
import numpy as np

# 300 MB binary blobs × 10 = 3 GB of total data
blob = b'x' * (300 * 1024 * 1024)
data = [blob] * 10
a = (np.array(data))

# test conversion to pyarrow array
print(type(pa.array(a))) # -> pyarrow.lib.ChunkedArray

indices = np.array(list(range(10)))

# this throws an error, even if a is a legit numpy array
pa.DictionaryArray.from_arrays(indices, a)

'''
------- RESULT ----------
Traceback (most recent call last):
File "", line 1, in
File "pyarrow\\array.pxi", line 4091, in pyarrow.lib.DictionaryArray.from_arrays
TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array
'''
``

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.