apache / apache/arrow

Are non-nullable null types (`pyarrow.Field<name: null not null>`) allowed?

Open
#41,308 0 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The pyarrow constructor prevents such a type:

```python
>>> import pyarrow as pa
>>> pa.field("name", pa.null(), nullable=False) # not allowed
```

raises

```
Traceback (most recent call last):
File "", line 1, in
File "pyarrow/types.pxi", line 2266, in pyarrow.lib.field
ValueError: A null type field may not be non-nullable
```

but there's a way around it, perhaps unintended:

```python
>>> pa.field("name", pa.null()).with_nullable(False) # equivalent, but allowed
pyarrow.Field
```

On the other hand, data created with this type can't be written to Parquet:

```
...
File "/home/jpivarski/irishep/awkward/src/awkward/operations/ak_to_parquet.py", line 416, in _impl
with pyarrow_parquet.ParquetWriter(
File "/home/jpivarski/mambaforge/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1010, in __init__
self.writer = _parquet.ParquetWriter(
File "pyarrow/_parquet.pyx", line 2157, in pyarrow._parquet.ParquetWriter.__cinit__
File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: NullType Arrow field must be nullable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "", line 1, in
File "/home/jpivarski/irishep/awkward/src/awkward/_dispatch.py", line 38, in dispatch
with OperationErrorContext(name, args, kwargs):
File "/home/jpivarski/irishep/awkward/src/awkward/_errors.py", line 85, in __exit__
self.handle_exception(exception_type, exception_value)
File "/home/jpivarski/irishep/awkward/src/awkward/_errors.py", line 95, in handle_exception
raise self.decorate_exception(cls, exception)
pyarrow.lib.ArrowInvalid: NullType Arrow field must be nullable
```

On the other, other hand, the specification is silent on the issue:

https://arrow.apache.org/docs/format/Columnar.html#null-layout

This came up in the context of Awkward Array conversions into Arrow and Parquet (through pyarrow), scikit-hep/awkward#2340. We can make this possibly-invalid Arrow type with

```python
>>> import awkward as ak
>>> arrow_array1 = ak.to_arrow(ak.Array([]), extensionarray=False, list_to32=True)
>>> arrow_array1

0 nulls
>>> arrow_array1.type
DataType(null)
```

and

```python
>>> arrow_array2 = ak.to_arrow(ak.Array([[], [], []]), extensionarray=False, list_to32=True)
>>> arrow_array2

[
0 nulls,
0 nulls,
0 nulls
]
>>> arrow_array2.type
ListType(list)
```

but get the stack trace above if we try to convert these arrays into Parquet. (Awkward Arrays default to non-nullable, and the list array conversion to Arrow uses `with_nullable`, so we may have found this unintended back-door to creating the non-nullable null type.)

Right now, we're developing a work-around to avoid creating Arrow arrays with non-nullable null types, assuming that it is invalid, as the error messages say. But...

**Is it invalid?** If so, the `with_nullable` back-door should be closed, and this fact should be documented in the specification.

**Is it valid?** If so, then the normal constructor should be able to construct it, and it should be writable to Parquet.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.