apache / apache/arrow

[C++] is_null(nan_is_null=true) ignores NaN and null values in dictionary-encoded float arrays

Open
#51,127 0 comments 0 reactions 1 assignee Claimed by @shoemoney View on GitHub
Component: C++
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, or platform.

`is_null(arr, nan_is_null=True)` does not mark NaN values as null when `arr` is dictionary-encoded with floating-point dictionary values.

`IsNullExec` decides whether to take the NaN path by looking at the top-level type id. For a dictionary-encoded array that id is `DICTIONARY`, so the floating-point check is skipped entirely and NaN entries in the dictionary are never marked null. The same call on the decoded (non-dictionary) array returns the expected result.

Related: null *values* inside the dictionary (as opposed to null indices) are also not reflected, so a valid index pointing at a null dictionary slot is reported as non-null.

Discussion and a first attempt at a fix are in #51000; @pitrou suggested computing `is_null` on the dictionary values, `take`-ing by the indices, and OR-ing the result with the index validity bitmap, which covers both the NaN case and null dictionary values in one pass.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.