apache / apache/arrow-rs

Use `nullable` marker instead of `non-null`

Open
#10,657 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

## Propsal
The `Display` of `Datatype` currently adds `non-null` markers for fields with `nullability=false`.
I want to instead mark the nullable fields with a `nullable` marker:

Proposed change:
* `List(non-null Uint8)` -> `List(Uint8)`
* `List(Uint8)` -> `List(nullable Uint8)`

## Motivation
### `nullable` is a concept in arrow-rs
In the source code, the term `nullable` is ubiquitous, but NOT `non-null`. For instance:

* https://docs.rs/arrow/latest/arrow/array/trait.Array.html#method.is_nullable
* https://docs.rs/arrow/latest/arrow/datatypes/struct.Field.html#method.is_nullable
* https://docs.rs/arrow/latest/arrow/ffi/struct.FFI_ArrowSchema.html#method.nullable

### `non-null` has a negation
`non-null` describes what something _isn't_. This is completely ~backwards~ non-forwards. We should describe what things _are_, rather than what they _aren't_.

### Nullability as a type
Modern languages recognizes that the ability to assign `null` (or `nil` or `None`…) to something is an _addition_ to its type. In Rust, `Option` marks something as "nullable". In haskell it is `Maybe`. The point is: if something can be _absent_ we _add more text to the type_. That is also what a `nullable` prefix would do.

## Migration
One big problem is backwards compatibility. In particular: how should `DataType::from_str` handle old strings? When parsing `"List(Uint8)"` from an old Arrow version, we should set `nullable=true`. But when parsing new strings, `"List(Uint8)"` means `nullable=false`.

### Solution A: break compatibility
The simple solution: break compatibility of the `Display/parse` of `DataType`. We do major releases anyway, so that's fine. Also: there are so many `DataType`s we still don't properly parse.

### Solution B: add some other marker to distinguish
Maybe we change `(…)` into `<…>`, so that going forward we write `List`. That way we can tell old string from new strings.

Contributor guide

Open the contributing guide

Research direction

Start by locating the DataType Display and from_str implementations, then review how nullability is represented in their current string forms. Resolve the migration choice described in the issue and update the formatting and parsing behavior consistently, with coverage for the old and proposed forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.