[Python] Can't provide Arrow array when filling nulls of a fixed size list array
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Suppose you have a fixed-sized list array with null values:
```py
array = pa.nulls(3, pa.list_(pa.float64(), 2))
print(array)
# [
# null,
# null,
# null
#]
```
How do you fill the nulls? Like, maybe I want it to look like `[[null, null], [null, null], [null, null]]`. Or even `[[0, 0], [0, 0], [0, 0]]`.
You can't call `fill_null` with a pyarrow array value, because the arrays don't have an '`as_py`' method:
```py
array.fill_null(pa.array([0.0, 0.0]))
```
```
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/array.pxi", line 1296, in pyarrow.lib.Array.fill_null
File "/Users/swnelson/scratch/.arrow-venv/lib/python3.11/site-packages/pyarrow/compute.py", line 523, in fill_null
fill_value = pa.scalar(fill_value.as_py(), type=values.type)
^^^^^^^^^^^^^^^^
AttributeError: 'pyarrow.lib.DoubleArray' object has no attribute 'as_py'
```
But you _can_ do it with a plain old Python list!
```py
array.fill_null([0.0, 0.0])
```
```
[
[
0,
0
],
[
0,
0
],
[
0,
0
]
]
```
This seems like an oversight. If I give an array of the right shape, surely it should be permitted.
### Component(s)
Python
Contributor guide
Research direction
Run the provided Python reproduction with a fixed-size list array and compare passing a Python list with passing a PyArrow array. Trace the fill_null path through pyarrow/array.pxi and the compute.py location shown in the traceback. Done means an appropriately shaped PyArrow array can fill the nulls without the as_py error, with regression coverage for the reported cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100