NVIDIA / NVIDIA/cudf

[QST] Difference in PQ reader output with AST filters if there are `nulls` in predicate col(s)?

Open
#18,291 0 comments 0 reactions 0 assignees View on GitHub
feature request Python question
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**What is your question?**

When reading a Parquet file with AST filter(s), cuDF removes all rows with `nulls` in **any** of the predicate column(s) whereas pyArrow keeps them. I tried to look for an existing issue for this difference in behavior but couldn't find one. Should we create one or convert this into a bug report? Here's a simple reproducer:

Maybe related #17142

```python
import cudf
import pyarrow.parquet as pq
from cudf.testing import assert_eq
from io import BytesIO

filters = [[("a", ">=", 2)], [("b", "<=", 5.0)]]

def test_filter(df):
buf = BytesIO()
df.to_parquet(buf)
df = cudf.read_parquet(buf, filters=filters).to_pandas()
df2 = pq.read_table(buf, filters=filters).to_pandas()
assert_eq(df, df2)

# no nulls
df = cudf.DataFrame(
{
"a": [1, 2, 3, 4, 5, 6],
"b": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0],
"c": ["a", "b", "c", "d", "e", "f"],
}
)
test_filter(df) # ok

# with nulls
df = cudf.DataFrame(
{
"a": [1, 2, None, 4, None, 6],
"b": [1.0, 2.0, 3.0, 4.0, 5.0, None],
"c": ["a", "b", "c", "d", None, "f"],
}
)
test_filter(df) # error
```

Output tables from cuDF and pyArrow respectively:

```bash
a b c
0 1 1.0 a
1 2 2.0 b
2 4 4.0 d

a b c
0 1 1.0 a
1 2 2.0 b
2 3.0 c
3 4 4.0 d
4 5.0
5 6 f
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.