NVIDIA / NVIDIA/cudf

[FEA] Verify that PQ bloom filter RG pruning actually happens in tests

Open
#23,620 0 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

All existing bloom-filter tests only catches a filter that prunes *too much* (rows would go missing) but never one that prunes *nothing*.

We cannot assert it in C++, since libcudf's Parquet writer cannot emit bloom filters, so a test cannot construct its own input. In Python we have committed fixtures, but `cudf.read_parquet` returns only a `DataFrame`, so the counter never reaches the test. And we don't want to work around that by calling pylibcudf directly from cuDF's test suite, [that file should stick to cuDF Python APIs](https://github.com/rapidsai/cudf/pull/22861#discussion_r3751655745).

**Describe the solution you'd like**

Add bloom filter write support to libcudf's Parquet writer.

The verification then becomes trivial in C++, since the counters are already exposed on `table_with_metadata` and already asserted in `cpp/tests/io/parquet_reader_test.cpp`. A test would write a file with a bloom filter, then assert that a present value keeps the row groups while an absent value prunes them to zero. The in-range part is what statistics cannot prune, so it isolates the bloom filter's contribution.

**Describe alternatives you've considered**

- Embedding a hexdump of an externally written file in the C++ test: rejected as unmaintainable.
- Committing binary fixtures and testing from Python: the current state, but cannot assert the counter.
- Exposing the counters through cuDF's Python API: new public API surface added only for testing.
- Adding pylibcudf tests that read cuDF's fixtures: adding fragile cross-package-path.
- Raising the pyarrow pin to ≥24 and generating fixtures at runtime: solves fixture generation but a length-absent header still needs a committed file.

**Additional context**

Bloom-filter fixtures live in `python/cudf/cudf/tests/data/parquet/` (`bloom_filter_alignment*.parquet`, `data_index_bloom_encoding_*.parquet`, `mixed_card_ndv_*_bf_*.parquet`).

Related: #22901 (bloom filter fetch and header-parsing fix), #23393 (Python bindings and pytest coverage for the hybrid scan reader), #23515 (length-absent bloom filters), #22861 (where the hexdump-based C++ test was removed in favor of pytests).

Contributor guide

Open the contributing guide

Research direction

Start with the bloom-filter fixtures in python/cudf/cudf/tests/data/parquet/ and the existing assertions in cpp/tests/io/parquet_reader_test.cpp. Trace how the Parquet writer produces table_with_metadata and identify the writer entry points needed for bloom-filter output. Done means a C++ test writes a bloom-filter file and verifies that a present in-range value retains row groups while an absent value prunes them to zero.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.