apache / apache/arrow-rs

Parquet decoding/pushdown performance improvements

Open
#9,589 2 comments 0 reactions 0 assignees View on GitHub
parquet performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

## Summary

Umbrella issue tracking a set of related parquet decoding performance improvements.

## Issues

- [ ] #9579 - Avoid unnecessary buffer zero-fill in Snappy decompression
- [x] #9587 - Pre-reserve output capacity in ByteView/ByteArray dictionary decoding
- [x] #9580 - Reduce per-byte overhead in VLQ integer decoding
- [ ] #9591 - Filter pushdown selectivity threshold
- [ ] #9581 - Use branchless index clamping and add `get_batch_direct` to RleDecoder
- [ ] #9582 - Fuse RLE decoding and view gathering for StringView dictionary decoding
- [ ] #9588 - Dictionary page pruning for row filter predicates

Contributor guide

Open the contributing guide

Research direction

This is an umbrella issue tracking Parquet performance work through #9579, #9591, #9581, #9582, and #9588; the checked items are already complete. Start by reading the specific unchecked issue you want to take on, then verify its stated performance improvement and update this checklist when that work is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.