Parquet decoding/pushdown performance improvements
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
## Summary
Umbrella issue tracking a set of related parquet decoding performance improvements.
## Issues
- [ ] #9579 - Avoid unnecessary buffer zero-fill in Snappy decompression
- [x] #9587 - Pre-reserve output capacity in ByteView/ByteArray dictionary decoding
- [x] #9580 - Reduce per-byte overhead in VLQ integer decoding
- [ ] #9591 - Filter pushdown selectivity threshold
- [ ] #9581 - Use branchless index clamping and add `get_batch_direct` to RleDecoder
- [ ] #9582 - Fuse RLE decoding and view gathering for StringView dictionary decoding
- [ ] #9588 - Dictionary page pruning for row filter predicates
Contributor guide
Research direction
This is an umbrella issue tracking Parquet performance work through #9579, #9591, #9581, #9582, and #9588; the checked items are already complete. Start by reading the specific unchecked issue you want to take on, then verify its stated performance improvement and update this checklist when that work is complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100