NVIDIA / NVIDIA/cudf

[BUG] read_parquet/read_orc with filters do not filter specific rows

Open
#12,512 2 comments 0 reactions 0 assignees View on GitHub
0 - Backlog bug cuIO feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
When using `cudf.read_parquet` or `read_orc` with the filters argument to filter out rows based on certain predicates, the methods today just filter out reading row groups (or stripes) that can be completely eliminated based on the given condition, but does return all rows from the read row groups without applying the given filters again. This behavior can be confusing to users assuming that all the relevant data has already been filtered out and is contrary to how dask, dask-cuDF and PyArrow behave today.

Example:

Data:
```
Col Name: A
Row Group 0: 1,5,1
Row Group 1: 5,5,5
```
```python
cudf.read_parquet("data", filters=[('a','!=',5)])
```
Would return 1 , 5, 1 which is all elements from RG0 (RG1 gets filtered out).
Expected output would be 1,1

**Steps/Code to reproduce bug**
```python
df = cudf.DataFrame()

In [6]: df["a"] = [1,5]*2500 + [5]*5000

In [7]: df.to_parquet("rg_test.parquet", row_group_size_rows=5000)

In [8]: cudf.read_parquet("rg_test.parquet")
[10000 rows x 1 columns]

In [9]: cudf.read_parquet("rg_test.parquet", filters=[("a", "!=", 5)])
[5000 rows x 1 columns]
```

**Expected behavior**
The 5's from row group 0 also get filtered returning only 1's, which is inline with how pyarrow, dask/dask-cudf return return the result.

**Environment overview (please complete the following information)**
- Environment location: bare-metal
- Method of cuDF install: conda
- If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.