apache / apache/iceberg-rust

Support Parquet bloom filter pruning on read

Closed
#2,841 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

### Is your feature request related to a problem or challenge?

Right now `ArrowReader` only prunes row groups using min/max stats. But for lookups like `trace_id = '...'` (where values are basically random 128-bit ids), min/max overlaps every row group and nothing gets pruned. The files do have bloom filters on those columns, the reader just never looks at them.

Writing bloom filters already works, you can turn them on per column via `WriterProperties`. It's purely the read side that's missing.

### Describe the solution you'd like

When a predicate has an eq or IN check on a column with a `bloom_filter_offset`, load the filter and skip any row group that comes back negative. Might make sense as an opt-in on the scan builder, since it's an extra read per row group checked.

#2398 already did this, but it got closed by the stale bot before anyone reviewed it. @dannycjones suggested to file a feature request, so here it is.

A few other places already do this if it helps: Iceberg Java has `ParquetBloomRowGroupFilter`, DataFusion does it by default (`bloom_filter_on_read`), and both Trino and DuckDB check bloom filters on read too.

### Willingness to contribute

None (edit: I could definitely test it)

Contributor guide

Open the contributing guide

Research direction

Start with ArrowReader's existing min/max row-group pruning and the scan builder, then trace how Parquet metadata exposes bloom_filter_offset. Check how eq and IN predicates are represented and how WriterProperties enables bloom filters. Done means supported predicates skip row groups when the bloom filter is negative, with an opt-in scan setting if needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.