apache / apache/arrow-rs

Consider adding BloomFilter reading support to `ParquetMetadataReader`

Open
#6,514 4 comments 2 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Parquet now has the wonderful [`ParquetMetaDataReader`](https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataReader.html) structure from @adriangb and @etseidl

This handles reading the footer metadata as well as the page indexes.

@progval noted in https://github.com/apache/arrow-rs/pull/6505#discussion_r1787879839 that BloomFilters are similiar to the PageIndex, but are not currently read/written by the ParquetMetaDataReader

**Describe the solution you'd like**
I would like to be able to configure the ParquetMetaDataReader (and writer) to read BloomFilters as well

**Describe alternatives you've considered**
This might look something like

```rust
// read parquet metadata including page indexes
let file = open_parquet_file("some_path.parquet");
let mut reader = ParquetMetaDataReader::new()
.with_bloom_filters(true);
reader.try_parse(&file).unwrap();
let metadata = reader.finish().unwrap();
// Somehow get access to the bloom filters (not sure what that API would look like)

```
**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the ParquetMetaDataReader entry point and the linked Arrow PR discussion about BloomFilters and PageIndex support. Trace how try_parse and finish expose metadata, then determine the reader and writer configuration and BloomFilter access needed for the feature to be complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.