Ability to pass arbitrary pages to SerializedPageReader
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We want ability to read an arbitrary page in a column chunk, [`SerializedPageReader`](https://github.com/apache/arrow-rs/blob/9413cd3ffdccdc529e44b7aa9d77c9565f7ecaca/parquet/src/file/serialized_reader.rs#L540) has _almost_ everything we need to. The only catch is that there is an implicit constraint that we have to pass the first data page to the `page_locations` argument of the `new_with_properties` constructor. This is fine but it makes working with this reader less ergonomic (you have to skip a page to get to the page you actually want to read).
**Describe the solution you'd like**
1. Lift the implicit constraint that we have to pass the first data page to the `page_locations` argument. Internally, we can read page_index (if available) to infer the dictionary page size;
2. Add an example or a test on how to use this API.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start in parquet/src/file/serialized_reader.rs at SerializedPageReader::new_with_properties and trace how page_locations and page-index information are handled. Add an example or test covering construction from an arbitrary page, with completion shown by successfully reading the requested page without first skipping a page.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100