[BUG] read_parquet/read_orc with filters do not filter specific rows
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
When using `cudf.read_parquet` or `read_orc` with the filters argument to filter out rows based on certain predicates, the methods today just filter out reading row groups (or stripes) that can be completely eliminated based on the given condition, but does return all rows from the read row groups without applying the given filters again. This behavior can be confusing to users assuming that all the relevant data has already been filtered out and is contrary to how dask, dask-cuDF and PyArrow behave today.
Example:
Data:
```
Col Name: A
Row Group 0: 1,5,1
Row Group 1: 5,5,5
```
```python
cudf.read_parquet("data", filters=[('a','!=',5)])
```
Would return 1 , 5, 1 which is all elements from RG0 (RG1 gets filtered out).
Expected output would be 1,1
**Steps/Code to reproduce bug**
```python
df = cudf.DataFrame()
In [6]: df["a"] = [1,5]*2500 + [5]*5000
In [7]: df.to_parquet("rg_test.parquet", row_group_size_rows=5000)
In [8]: cudf.read_parquet("rg_test.parquet")
[10000 rows x 1 columns]
In [9]: cudf.read_parquet("rg_test.parquet", filters=[("a", "!=", 5)])
[5000 rows x 1 columns]
```
**Expected behavior**
The 5's from row group 0 also get filtered returning only 1's, which is inline with how pyarrow, dask/dask-cudf return return the result.
**Environment overview (please complete the following information)**
- Environment location: bare-metal
- Method of cuDF install: conda
- If method of install is [Docker], provide `docker pull` & `docker run` commands used
**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details
**Additional context**
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.