apache / apache/arrow-rs

Physical null and logical null are confusing concepts

Open
#4,840 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The arrow-rs library (now) makes a distinction between physical nulls and logical nulls as the same distinction is made in the Arrow specification (though the terms physical and logical nulls are not used, to my knoweldge)

The issue is that for certain array types computing if an element is very fast (consult a pre-existing bitmap) but for others can be quite slow (e.g. a dictionary where both the keys and values must be consulted for nullness)

The method named `Array::is_null` returns the (fast) physical nullness, but is deeply confusing for for certain types -- see https://github.com/apache/arrow-rs/issues/4835 and https://github.com/apache/arrow-rs/pull/4838#discussion_r1330002357 from @crepererum.

We have tried to clarify the difference in https://github.com/apache/arrow-rs/pull/4838 but it is still confusing

**Describe the solution you'd like**
I am not sure -- @crepererum suggests in https://github.com/apache/arrow-rs/pull/4838#discussion_r1330002357

> I would argue that at least this method should be called is_physical_null to force users to think about what kind of null they want, instead of tricking them into using the wrong implicit default for their use case.

However, there are downsides to this too

**Describe alternatives you've considered**
The documentation changes may be enough, but I think the issue is important enough to track here

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with Array::is_null and the discussion in issues 4835 and pull request 4838, especially the linked review comment. Compare the current physical-null behavior with the Arrow specification and affected array types such as dictionaries. This issue is complete only after the project agrees on whether an API or documentation change is needed and records that decision with appropriate tests or documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.