NVIDIA / NVIDIA/cudf

[FEA] Allow Hybrid Scan PQ reader to prune row groups with length-absent bloom filters

Open
#23,515 4 comments 0 reactions 0 assignees View on GitHub
0 - Backlog cuIO feature request good first issue
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem?**

`parquet::fetch_bloom_filters_to_device` treats any size-zero range as “no filter” and skips it. Consequently, these chunks are never bloom-pruned. This behavior is correct but suboptimal: pruning is silently lost because the device caster conservatively returns membership `true` for empty spans in `bloom_filter_reader.cu`.

The default `read_parquet` path already handles this case. `read_bloom_filters` substitutes a speculative 256-byte initial read, clamped to the datasource size, using `speculative_read_size` in:

https://github.com/rapidsai/cudf/blob/3b3cea4d9669b9db1e35769ea727514eac3cb3e7/cpp/src/io/parquet/bloom_filter_reader.cu#L363-L366

**Describe the solution you’d like**

Handle this case in `fetch_bloom_filters_to_device_impl` in `cpp/src/io/parquet/io_utils/parquet_io_utils.cpp`

This function owns the datasources and can therefore clamp reads safely. The hybrid-scan metadata layer cannot do so because it has no access to the datasources.

**Phase 1:**

Distinguish between:

- `{offset > 0, 0}`: A filter is present, but its length is unknown.
- `{0, 0}`: No filter is present.

An offset of zero is impossible for a real filter because Parquet files begin with `PAR1`.

For `{offset > 0, 0}`, use the following initial read size:

```cpp
std::min(speculative_read_size, datasource.size() - offset)
```

Here, `speculative_read_size` is 256 bytes.

**Phases 2–3:**

No changes are needed:

- Header parsing already recovers the actual bitset size.
- The deferred read already fetches any data not covered by the speculative read.

Move the `speculative_read_size` constant next to the fetch helper so it can be shared.

**Backward compatibility**

- The default reader never sends `{offset > 0, 0}` because it substitutes the speculative read size upstream.
- Handling of `{0, 0}` remains unchanged.

**Addition context**

This metadata is produced by older Spark/parquet-mr writers. One example is Apache’s `data_index_bloom_encoding_stats.parquet`.

Contributor guide

Open the contributing guide

Research direction

Start in cpp/src/io/parquet/io_utils/parquet_io_utils.cpp at fetch_bloom_filters_to_device_impl, then compare the speculative-read handling in cpp/src/io/parquet/bloom_filter_reader.cu. Preserve {0, 0} as no filter, use a clamped initial read for {offset > 0, 0}, and confirm existing header parsing and deferred reads still cover the full filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.