apache / apache/arrow-rs

Enable access to column dictionaries in async reader

Open
#9,010 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Some databases, one example being Grafana Tempo, utilize column dictionaries as makeshift column indexes, to improve filtering speed ad-hoc. Checking if low-cardinality value is present in dictionary allows to effectively pre-filter data by skipping whole row group

**Describe the solution you'd like**

Add API to `ParquetRecordBatchStreamBuilder` that allows to inspect contents of the dictionary

**Describe alternatives you've considered**

Column indexes have high expected size cost and are not always available (e.g. for legacy data)

**Additional context**

It is possible to access this information in `SerializedFileReader` already by using "peekable" page iterator

Contributor guide

Open the contributing guide

Research direction

Start with the ParquetRecordBatchStreamBuilder API and compare its async reader flow with the existing SerializedFileReader access through the peekable page iterator. Trace how dictionary contents are exposed there and determine the corresponding async entry point. Done means callers can inspect column dictionaries through the async builder for pre-filtering, with behavior consistent with the existing reader.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.