apache / apache/arrow

null structarrays are poorly handled by cast

Open
#37,072 2 comments 0 reactions 1 assignee Claimed by @felipecrv View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

This code should work:
```py
import pyarrow as pa
struct_type = pa.struct([pa.field("x", pa.int32(), nullable=False)])
nulls = pa.nulls(5, struct_type)

# The following is an error:
nulls = nulls.cast(struct_type)
```

The error message is:
```
ArrowInvalid: Can't view array of type struct as struct: nulls in input cannot be viewed as non-nullable
```

Indeed, if we `print(nulls)`, it contains null values in the non-nullable field `x`:
```
-- is_valid:
[
false,
false,
false,
false,
false
]
-- child 0 type: int32
[
null,
null,
null,
null,
null
]
```

But those are all invalid at the top-level _anyway_, so there's no reason `cast` ought to care. Either that, or it should be impossible to call `pa.nulls` on a struct with a non-nullable field anywhere in its hierarchy of fields, but that seems wrong too. That would imply that if _any_ field is non-nullable then the whole struct would be non-nullable, which clearly is not the intent. You should be able to have a null struct with non-nullable fields.

Ultimately, this is a C++ issue; Python is merely calling those functions.

### Version

12.0.1

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.