apache / apache/arrow

[C++] `is_null` kernel does not propagate `nan_is_null` for nested types

Open
#50,889 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

`b` is just a dictionary-encoded copy of `a`, so the compute kernel should have the same output. But it dosen't because `nan_is_null` is ignored when looking up dictionary values. This will involve changes to `SetLogicalNullBits` (see #49889). Also consider updating at `ArraySpan::ComputeLogicalNullCount`.

```
>>> a = pa.array([1.0, 1.5, 1.0, math.nan])
>>> b = pc.dictionary_encode(a)
>>> pc.is_null(a, nan_is_null=True)

[
false,
false,
false,
true
]
>>> pc.is_null(b, nan_is_null=True)

[
false,
false,
false,
false
]
```

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Reproduce the Python examples with is_null, dictionary_encode, and nan_is_null=True, then trace the C++ compute kernel through SetLogicalNullBits for nested and dictionary-encoded values. Also inspect ArraySpan::ComputeLogicalNullCount as suggested. Done means dictionary-encoded input produces the same nullness output as the original array, including NaN handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.