[FEA] Verify that PQ bloom filter RG pruning actually happens in tests
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
All existing bloom-filter tests only catches a filter that prunes *too much* (rows would go missing) but never one that prunes *nothing*.
We cannot assert it in C++, since libcudf's Parquet writer cannot emit bloom filters, so a test cannot construct its own input. In Python we have committed fixtures, but `cudf.read_parquet` returns only a `DataFrame`, so the counter never reaches the test. And we don't want to work around that by calling pylibcudf directly from cuDF's test suite, [that file should stick to cuDF Python APIs](https://github.com/rapidsai/cudf/pull/22861#discussion_r3751655745).
**Describe the solution you'd like**
Add bloom filter write support to libcudf's Parquet writer.
The verification then becomes trivial in C++, since the counters are already exposed on `table_with_metadata` and already asserted in `cpp/tests/io/parquet_reader_test.cpp`. A test would write a file with a bloom filter, then assert that a present value keeps the row groups while an absent value prunes them to zero. The in-range part is what statistics cannot prune, so it isolates the bloom filter's contribution.
**Describe alternatives you've considered**
- Embedding a hexdump of an externally written file in the C++ test: rejected as unmaintainable.
- Committing binary fixtures and testing from Python: the current state, but cannot assert the counter.
- Exposing the counters through cuDF's Python API: new public API surface added only for testing.
- Adding pylibcudf tests that read cuDF's fixtures: adding fragile cross-package-path.
- Raising the pyarrow pin to ≥24 and generating fixtures at runtime: solves fixture generation but a length-absent header still needs a committed file.
**Additional context**
Bloom-filter fixtures live in `python/cudf/cudf/tests/data/parquet/` (`bloom_filter_alignment*.parquet`, `data_index_bloom_encoding_*.parquet`, `mixed_card_ndv_*_bf_*.parquet`).
Related: #22901 (bloom filter fetch and header-parsing fix), #23393 (Python bindings and pytest coverage for the hybrid scan reader), #23515 (length-absent bloom filters), #22861 (where the hexdump-based C++ test was removed in favor of pytests).
Contributor guide
Research direction
Start with the bloom-filter fixtures in python/cudf/cudf/tests/data/parquet/ and the existing assertions in cpp/tests/io/parquet_reader_test.cpp. Trace how the Parquet writer produces table_with_metadata and identify the writer entry points needed for bloom-filter output. Done means a C++ test writes a bloom-filter file and verifies that a present in-range value retains row groups while an absent value prunes them to zero.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100