apache / apache/arrow

[Python] pyarrow.nulls doesn't return a valid array for non-nullable nested fields

Open
#43,358 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Say I have a struct with non-nullable fields, If I try to create an empty array of this struct, the array is not valid.

```
import pyarrow as pa
import pytest

def test_bad_nulls():
struct_type = pa.struct(
[
pa.field("field_1", pa.string(), nullable=False),
pa.field("field_2", pa.int64(), nullable=False),
]
)

with pytest.raises(
pa.ArrowInvalid,
match=r"Can't view array of type "
r"struct\ as "
r"struct\"
r"\: nulls in input cannot be viewed as non\-nullable",
):
pa.nulls(size=10, type=struct_type).cast(struct_type)

actual_nulls = pa.StructArray.from_arrays(
[pa.array([""] * 10), pa.array([0] * 10)],
fields=struct_type,
mask=pa.array([True] * 10),
)
actual_nulls.cast(struct_type)
assert actual_nulls.null_count == 10
assert actual_nulls.to_pylist() == [None] * 10
```

I guess it's because `pyarrow.nulls` initialize the nested arrays to nulls, but they are not nullable so they should be initialized to default values.

Alternatively, the cast / validate method should not check for the validity of the nested arrays, when the parent struct array isn't valid/is null for this element. Though I don't think this is a good idea, because if the nested arrays are reused outside of the struct, then they become invalid.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.