apache / apache/arrow

[Python] StructArray.from_arrays does not work well with FixedSizeList-typed data

Open
#38,026 2 comments 0 reactions 1 assignee Claimed by @spenczar View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Pyarrow's `StructArray.from_arrays` constructor will reject valid non-arrow data (for example a Python list-of-lists or Pandas Series) if the struct field specifies a fixed-length list.

Here's an example - first a bit of setup:
```py
import pyarrow as pa
import pandas as pd
dtype = pa.list_(pa.int64(), 2)
data_list = [[1, 2], [3, 4]]
data_pd_series = pd.Series(data_list)
data_array = pa.array(data_list, dtype)
```

Now, calling `pa.StructArray.from_arrays(foo, fields=[pa.field("x", dtype)])` should work regardless of whether `foo` is `data_list`, or `data_pd_series`, or `data_array`. But no:

```py
>>> sa = pa.StructArray.from_arrays([data_list], fields=[pa.field("x", dtype)])
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/array.pxi", line 2886, in pyarrow.lib.StructArray.from_arrays
File "pyarrow/array.pxi", line 1596, in pyarrow.lib.Array.validate
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Struct child array #0 does not match type field: list vs fixed_size_list[2]
>>> sa = pa.StructArray.from_arrays([data_pd_series], fields=[pa.field("x", dtype)])
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/array.pxi", line 2886, in pyarrow.lib.StructArray.from_arrays
File "pyarrow/array.pxi", line 1596, in pyarrow.lib.Array.validate
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Struct child array #0 does not match type field: list vs fixed_size_list[2]
>>>
>>> sa = pa.StructArray.from_arrays([data_array], fields=[pa.field("x", dtype)])
```

It appears that `pa.StructArray.from_arrays` casts non-pyarrow types, and _then_ validates against the schema without attempting another cast, while `pa.array` works differently somehow.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.