[Python] pyarrow.nulls doesn't return a valid array for non-nullable nested fields
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Say I have a struct with non-nullable fields, If I try to create an empty array of this struct, the array is not valid.
```
import pyarrow as pa
import pytest
def test_bad_nulls():
struct_type = pa.struct(
[
pa.field("field_1", pa.string(), nullable=False),
pa.field("field_2", pa.int64(), nullable=False),
]
)
with pytest.raises(
pa.ArrowInvalid,
match=r"Can't view array of type "
r"struct\ as "
r"struct\"
r"\: nulls in input cannot be viewed as non\-nullable",
):
pa.nulls(size=10, type=struct_type).cast(struct_type)
actual_nulls = pa.StructArray.from_arrays(
[pa.array([""] * 10), pa.array([0] * 10)],
fields=struct_type,
mask=pa.array([True] * 10),
)
actual_nulls.cast(struct_type)
assert actual_nulls.null_count == 10
assert actual_nulls.to_pylist() == [None] * 10
```
I guess it's because `pyarrow.nulls` initialize the nested arrays to nulls, but they are not nullable so they should be initialized to default values.
Alternatively, the cast / validate method should not check for the validity of the nested arrays, when the parent struct array isn't valid/is null for this element. Though I don't think this is a good idea, because if the nested arrays are reused outside of the struct, then they become invalid.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.