apache / apache/arrow

[C++][Python] pyarrow.compute.field("x").isin([0.0]) incorrectly filters out all rows when they all match

Open
#46,183 7 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Hi,

In some (rather specific) situations, the following filter expression: `pyarrow.compute.field("x").isin([0.0])` incorrectly filters out matching rows.

Here is a minimal reproducer:

```python
from tempfile import TemporaryDirectory

import pyarrow.compute
import pyarrow.dataset
import pyarrow.parquet

data = pyarrow.table({"x": [0.0]})

f0 = pyarrow.compute.field("x") == 0.0
f1 = pyarrow.compute.field("x").isin([0.0])

with TemporaryDirectory() as tmpdir:
pyarrow.parquet.write_to_dataset(data, tmpdir)

assert data == pyarrow.dataset.dataset(tmpdir).to_table()
assert data == pyarrow.dataset.dataset(tmpdir).filter(f0).to_table()
assert data == (actual := pyarrow.dataset.dataset(tmpdir).filter(f1).to_table()), actual
```

Output:

```
Traceback (most recent call last):
File "/tmp/tmp.iectDMOJmW/test.py", line 18, in
assert data == (actual := pyarrow.dataset.dataset(tmpdir).filter(f1).to_table()), actual
AssertionError: pyarrow.Table
x: double
----
x: [[]]
```

This does not happen if `x` contains other values that are not `0.0` ( `pyarrow.table({"x": [0.0, 1.0]})` --> ✅ | `pyarrow.table({"x": [0.0, 0.0, 0.0]})` --> 💥).

I am able to reproduce this bug for all released versions since v12.

### Component(s)

Python, C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.