[FEA] Allow Hybrid Scan PQ reader to prune row groups with length-absent bloom filters
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem?**
`parquet::fetch_bloom_filters_to_device` treats any size-zero range as “no filter” and skips it. Consequently, these chunks are never bloom-pruned. This behavior is correct but suboptimal: pruning is silently lost because the device caster conservatively returns membership `true` for empty spans in `bloom_filter_reader.cu`.
The default `read_parquet` path already handles this case. `read_bloom_filters` substitutes a speculative 256-byte initial read, clamped to the datasource size, using `speculative_read_size` in:
https://github.com/rapidsai/cudf/blob/3b3cea4d9669b9db1e35769ea727514eac3cb3e7/cpp/src/io/parquet/bloom_filter_reader.cu#L363-L366
**Describe the solution you’d like**
Handle this case in `fetch_bloom_filters_to_device_impl` in `cpp/src/io/parquet/io_utils/parquet_io_utils.cpp`
This function owns the datasources and can therefore clamp reads safely. The hybrid-scan metadata layer cannot do so because it has no access to the datasources.
**Phase 1:**
Distinguish between:
- `{offset > 0, 0}`: A filter is present, but its length is unknown.
- `{0, 0}`: No filter is present.
An offset of zero is impossible for a real filter because Parquet files begin with `PAR1`.
For `{offset > 0, 0}`, use the following initial read size:
```cpp
std::min(speculative_read_size, datasource.size() - offset)
```
Here, `speculative_read_size` is 256 bytes.
**Phases 2–3:**
No changes are needed:
- Header parsing already recovers the actual bitset size.
- The deferred read already fetches any data not covered by the speculative read.
Move the `speculative_read_size` constant next to the fetch helper so it can be shared.
**Backward compatibility**
- The default reader never sends `{offset > 0, 0}` because it substitutes the speculative read size upstream.
- Handling of `{0, 0}` remains unchanged.
**Addition context**
This metadata is produced by older Spark/parquet-mr writers. One example is Apache’s `data_index_bloom_encoding_stats.parquet`.
Contributor guide
Research direction
Start in cpp/src/io/parquet/io_utils/parquet_io_utils.cpp at fetch_bloom_filters_to_device_impl, then compare the speculative-read handling in cpp/src/io/parquet/bloom_filter_reader.cu. Preserve {0, 0} as no filter, use a clamped initial read for {offset > 0, 0}, and confirm existing header parsing and deferred reads still cover the full filter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100