Use `nullable` marker instead of `non-null`
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
## Propsal
The `Display` of `Datatype` currently adds `non-null` markers for fields with `nullability=false`.
I want to instead mark the nullable fields with a `nullable` marker:
Proposed change:
* `List(non-null Uint8)` -> `List(Uint8)`
* `List(Uint8)` -> `List(nullable Uint8)`
## Motivation
### `nullable` is a concept in arrow-rs
In the source code, the term `nullable` is ubiquitous, but NOT `non-null`. For instance:
* https://docs.rs/arrow/latest/arrow/array/trait.Array.html#method.is_nullable
* https://docs.rs/arrow/latest/arrow/datatypes/struct.Field.html#method.is_nullable
* https://docs.rs/arrow/latest/arrow/ffi/struct.FFI_ArrowSchema.html#method.nullable
### `non-null` has a negation
`non-null` describes what something _isn't_. This is completely ~backwards~ non-forwards. We should describe what things _are_, rather than what they _aren't_.
### Nullability as a type
Modern languages recognizes that the ability to assign `null` (or `nil` or `None`…) to something is an _addition_ to its type. In Rust, `Option` marks something as "nullable". In haskell it is `Maybe`. The point is: if something can be _absent_ we _add more text to the type_. That is also what a `nullable` prefix would do.
## Migration
One big problem is backwards compatibility. In particular: how should `DataType::from_str` handle old strings? When parsing `"List(Uint8)"` from an old Arrow version, we should set `nullable=true`. But when parsing new strings, `"List(Uint8)"` means `nullable=false`.
### Solution A: break compatibility
The simple solution: break compatibility of the `Display/parse` of `DataType`. We do major releases anyway, so that's fine. Also: there are so many `DataType`s we still don't properly parse.
### Solution B: add some other marker to distinguish
Maybe we change `(…)` into `<…>`, so that going forward we write `List`. That way we can tell old string from new strings.
Contributor guide
Research direction
Start by locating the DataType Display and from_str implementations, then review how nullability is represented in their current string forms. Resolve the migration choice described in the issue and update the formatting and parsing behavior consistently, with coverage for the old and proposed forms.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100