Enable access to column dictionaries in async reader
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Some databases, one example being Grafana Tempo, utilize column dictionaries as makeshift column indexes, to improve filtering speed ad-hoc. Checking if low-cardinality value is present in dictionary allows to effectively pre-filter data by skipping whole row group
**Describe the solution you'd like**
Add API to `ParquetRecordBatchStreamBuilder` that allows to inspect contents of the dictionary
**Describe alternatives you've considered**
Column indexes have high expected size cost and are not always available (e.g. for legacy data)
**Additional context**
It is possible to access this information in `SerializedFileReader` already by using "peekable" page iterator
Contributor guide
Research direction
Start with the ParquetRecordBatchStreamBuilder API and compare its async reader flow with the existing SerializedFileReader access through the peekable page iterator. Trace how dictionary contents are exposed there and determine the corresponding async entry point. Done means callers can inspect column dictionaries through the async builder for pre-filtering, with behavior consistent with the existing reader.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100