apache / apache/arrow

[Python] Array equals returns incorrectly if NaNs are in arrays

Open
#22,446 15 comments 0 reactions 0 assignees View on GitHub
Component: Python Priority: Critical Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 20h
Merged PRs (30d)
94

Description

```python

import numpy as np
import pyarrow as pa

data = [0, 1, np.nan, None, 4]

arr1 = pa.array(data)
arr2 = pa.array(data)

pa.Array.equals(arr1, arr2)
```

Unsure if this is expected behavior, but in Arrow 0.12.1 this returned `True` as compared to `False` in 0.14.1.

**Reporter**: [Keith Kraus](https://issues.apache.org/jira/browse/ARROW-6043) / @kkraus14
#### Related issues:
- [[Python] pyarrow array equals return False if there's nan ](https://github.com/apache/arrow/issues/29227) (duplicates)
- [[C++] Should NaN comparison return false or NaN/NA?](https://github.com/apache/arrow/issues/29038) (relates to)
#### PRs and other links:
- [GitHub Pull Request #8288](https://github.com/apache/arrow/pull/8288)

**Note**: *This issue was originally created as [ARROW-6043](https://issues.apache.org/jira/browse/ARROW-6043). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the Python example with the reported Arrow versions, then read Pull Request #8288 and the linked duplicate and comparison issues to understand the intended NaN equality semantics. Done means the behavior is decided, implemented consistently, and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, numpy, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.