[C++] is_null(nan_is_null=true) ignores NaN and null values in dictionary-encoded float arrays
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, or platform.
`is_null(arr, nan_is_null=True)` does not mark NaN values as null when `arr` is dictionary-encoded with floating-point dictionary values.
`IsNullExec` decides whether to take the NaN path by looking at the top-level type id. For a dictionary-encoded array that id is `DICTIONARY`, so the floating-point check is skipped entirely and NaN entries in the dictionary are never marked null. The same call on the decoded (non-dictionary) array returns the expected result.
Related: null *values* inside the dictionary (as opposed to null indices) are also not reflected, so a valid index pointing at a null dictionary slot is reported as non-null.
Discussion and a first attempt at a fix are in #51000; @pitrou suggested computing `is_null` on the dictionary values, `take`-ing by the indices, and OR-ing the result with the index validity bitmap, which covers both the NaN case and null dictionary values in one pass.
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.