Call `get_row_group_column_bloom_filter` in parallel
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
Whilst working on https://github.com/apache/iceberg-rust/pull/2398 we realized that calling `get_row_group_column_bloom_filter` in a loop isn't optimal.
DataFusion also does it the same way https://github.com/apache/datafusion/blob/55.0.0/datafusion/datasource-parquet/src/opener/mod.rs#L1266-L1292
https://github.com/apache/arrow-rs/pull/8462/changes has merged so we should follow up to improve this implementation.
Contributor guide
Research direction
Start by locating the implementation that calls get_row_group_column_bloom_filter and read the related Arrow and DataFusion examples linked in the issue. Update the implementation so the bloom-filter calls run in parallel, then run the relevant existing tests and confirm the behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100