apache / apache/arrow-rs

Ability to pass arbitrary pages to SerializedPageReader

Open
#5,940 6 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We want ability to read an arbitrary page in a column chunk, [`SerializedPageReader`](https://github.com/apache/arrow-rs/blob/9413cd3ffdccdc529e44b7aa9d77c9565f7ecaca/parquet/src/file/serialized_reader.rs#L540) has _almost_ everything we need to. The only catch is that there is an implicit constraint that we have to pass the first data page to the `page_locations` argument of the `new_with_properties` constructor. This is fine but it makes working with this reader less ergonomic (you have to skip a page to get to the page you actually want to read).

**Describe the solution you'd like**
1. Lift the implicit constraint that we have to pass the first data page to the `page_locations` argument. Internally, we can read page_index (if available) to infer the dictionary page size;
2. Add an example or a test on how to use this API.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start in parquet/src/file/serialized_reader.rs at SerializedPageReader::new_with_properties and trace how page_locations and page-index information are handled. Add an example or test covering construction from an arbitrary page, with completion shown by successfully reading the requested page without first skipping a page.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.