[QST] Difference in PQ reader output with AST filters if there are `nulls` in predicate col(s)?
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**What is your question?**
When reading a Parquet file with AST filter(s), cuDF removes all rows with `nulls` in **any** of the predicate column(s) whereas pyArrow keeps them. I tried to look for an existing issue for this difference in behavior but couldn't find one. Should we create one or convert this into a bug report? Here's a simple reproducer:
Maybe related #17142
```python
import cudf
import pyarrow.parquet as pq
from cudf.testing import assert_eq
from io import BytesIO
filters = [[("a", ">=", 2)], [("b", "<=", 5.0)]]
def test_filter(df):
buf = BytesIO()
df.to_parquet(buf)
df = cudf.read_parquet(buf, filters=filters).to_pandas()
df2 = pq.read_table(buf, filters=filters).to_pandas()
assert_eq(df, df2)
# no nulls
df = cudf.DataFrame(
{
"a": [1, 2, 3, 4, 5, 6],
"b": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0],
"c": ["a", "b", "c", "d", "e", "f"],
}
)
test_filter(df) # ok
# with nulls
df = cudf.DataFrame(
{
"a": [1, 2, None, 4, None, 6],
"b": [1.0, 2.0, 3.0, 4.0, 5.0, None],
"c": ["a", "b", "c", "d", None, "f"],
}
)
test_filter(df) # error
```
Output tables from cuDF and pyArrow respectively:
```bash
a b c
0 1 1.0 a
1 2 2.0 b
2 4 4.0 d
a b c
0 1 1.0 a
1 2 2.0 b
2 3.0 c
3 4 4.0 d
4 5.0
5 6 f
```
Contributor guide
Assessment
This issue has not been assessed yet.