[Parquet] PushDecoder: Add a peek API to support pre-fetching
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- part of https://github.com/apache/arrow-rs/issues/8000
Unlike streams of JSON / CSV, the data that the parquet reader needs next i is not easy to predict as it depends on the filters, the row groups, which columns are requested, etc.
Now that we have the initial PushDecoder in this PR
- https://github.com/apache/arrow-rs/pull/7997
We will be in the position to add an API for the decoder to communicate what data will be needed next
**Describe the solution you'd like**
I would like an API that allows users of the Parquet decoder to have more fine grained control over peeking
**Describe alternatives you've considered**
Here is an idea from @adriangb on https://github.com/apache/arrow-rs/pull/7997/files#r2444922393
> a method along the lines of try_peek()? It'd be cool if it returned some structure that allowed fine grained control of the peeking:
```rust
let max_ranges = 32;
let max_bytes = 1024 * 1024 * 32;
let mut current_bytes = 0;
let mut ranges = Vec::new();
let mut peek = decoder.peek()
loop {
match peek.next() {
PeekResult::Range(range) => {
ranges.push(range);
current_bytes += range.end - range.start;
if ranges.len() > max_ranges { break }
if current_bytes > max_bytes { break }
PeekResult::End { break }
}
}
```
Here is another potential API from the original ticket:
```rust
// Create a decoder for decoding parquet data as above
let mut decoder: ParquetDecoderBuilder = ...;
// As the decoder up from what data it will need, start prefetching data if desired
while let Some(pre_request) = decoder.peek_next_requests() {
// note that this is a peek and if we call peek again in the
// future, we may get a different set of pre_requests (for example
// if the decoder has applied a row filter and ruled out
// some row groups or data pages)
start_prefetch(pre_request);
}
```
// push data to the decoder as before, but hopefully the reader
// will have already prefetched some of the data
**Additional context**
Contributor guide
Research direction
Start with the initial PushDecoder implementation in PR 7997 and the parent design discussion in issue 8000. Compare the proposed try_peek and peek_next_requests interfaces, then define an API that supports bounded range or byte peeking for prefetching; done means the design is agreed and the decoder can communicate upcoming data requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100