apache / apache/arrow

[Python] Can't provide Arrow array when filling nulls of a fixed size list array

Open
#35,624 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Suppose you have a fixed-sized list array with null values:

```py
array = pa.nulls(3, pa.list_(pa.float64(), 2))
print(array)
# [
# null,
# null,
# null
#]
```

How do you fill the nulls? Like, maybe I want it to look like `[[null, null], [null, null], [null, null]]`. Or even `[[0, 0], [0, 0], [0, 0]]`.

You can't call `fill_null` with a pyarrow array value, because the arrays don't have an '`as_py`' method:
```py
array.fill_null(pa.array([0.0, 0.0]))
```
```
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/array.pxi", line 1296, in pyarrow.lib.Array.fill_null
File "/Users/swnelson/scratch/.arrow-venv/lib/python3.11/site-packages/pyarrow/compute.py", line 523, in fill_null
fill_value = pa.scalar(fill_value.as_py(), type=values.type)
^^^^^^^^^^^^^^^^
AttributeError: 'pyarrow.lib.DoubleArray' object has no attribute 'as_py'
```

But you _can_ do it with a plain old Python list!

```py
array.fill_null([0.0, 0.0])
```
```

[
[
0,
0
],
[
0,
0
],
[
0,
0
]
]
```

This seems like an oversight. If I give an array of the right shape, surely it should be permitted.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Run the provided Python reproduction with a fixed-size list array and compare passing a Python list with passing a PyArrow array. Trace the fill_null path through pyarrow/array.pxi and the compute.py location shown in the traceback. Done means an appropriately shaped PyArrow array can fill the nulls without the as_py error, with regression coverage for the reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.