NVIDIA / NVIDIA/cudf

[BUG] hybrid scan build_row_mask_with_page_index_stats should not fail if page index isn't found

Open
#20,833 3 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
I am trying to load a file with the hybrid reader, and when I call `build_row_mask_with_page_index_stats` it fails with:
```
cudf/cpp/src/io/parquet/experimental/page_index_filter.cu:853: Page pruning requires the Parquet page index for all output columns
```

I believe, instead we should ignore this and return an "all rows" mask like: https://github.com/rapidsai/cudf/issues/20832. Alternatively, if there is a world where if one column has a page index, that all must, we could make that the check "column A had the page index, but columns B, C, D... didn't". I do not know if this is the case.

**Steps/Code to reproduce bug**

Try to read a parquet file without page indices with the hybrid reader.

**Expected behavior**
An "all rows" mask is what I would expect. In spark, the filtering done in parquet is opportunistic (it's done to reduce IO). Spark always adds a filter after the scan, to actually filter the rows it wants, so if we get all rows from this step, we would be OK.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.