Expose public async bloom filter reader (metadata + `AsyncFileReader`)
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I'm integrating Parquet bloom filters into an async pruning pipeline and found a gap in the public API.
For current situation,
- There is a sync API:
`Sbbf::read_from_column_chunk(column_meta, reader)`
- There is an async method, but only on the async Arrow builder:
`ParquetRecordBatchStreamBuilder::get_row_group_column_bloom_filter(...)`
- The helper used internally to parse bloom filter headers is `pub(crate)`:
`chunk_read_bloom_filter_header_and_offset` (in `parquet::bloom_filter`)
If parquet crate only has `ParquetMetaData` + an `AsyncFileReader`, downstream applications can't read bloom filters without re‑implementing Parquet bloom header parsing.
This blocks async metadata‑only pruning libraries (like me) from using bloom filters safely and efficiently.
**Describe the solution you'd like**
Expose a public async bloom reader that mirrors the sync API:
```rust
pub async fn read_bloom_filter_async(
column_meta: &ColumnChunkMetaData,
reader: &mut R
) -> Result>;
```
This would:
- keep internal header parsing private
- allow async pruning without coupling to Arrow builder
- avoid duplicate parsing logic in downstream crates
- be backwards compatible (pure API addition)
*Alternative*
Make `chunk_read_bloom_filter_header_and_offset` public, but this is a low‑level parsing helper and would bake in more implementation detail.
Contributor guide
Research direction
Start in parquet::bloom_filter by reading chunk_read_bloom_filter_header_and_offset, then compare Sbbf::read_from_column_chunk with ParquetRecordBatchStreamBuilder::get_row_group_column_bloom_filter and the AsyncFileReader interface. Done means a public async reader accepts ColumnChunkMetaData and an AsyncFileReader, returns the requested bloom filter result, and keeps internal header parsing private.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100