apache / apache/arrow-rs

[Parquet] PushDecoder: Add a peek API to support pre-fetching

Open
#8,668 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- part of https://github.com/apache/arrow-rs/issues/8000

Unlike streams of JSON / CSV, the data that the parquet reader needs next i is not easy to predict as it depends on the filters, the row groups, which columns are requested, etc.

Now that we have the initial PushDecoder in this PR
- https://github.com/apache/arrow-rs/pull/7997

We will be in the position to add an API for the decoder to communicate what data will be needed next

**Describe the solution you'd like**
I would like an API that allows users of the Parquet decoder to have more fine grained control over peeking

**Describe alternatives you've considered**

Here is an idea from @adriangb on https://github.com/apache/arrow-rs/pull/7997/files#r2444922393

> a method along the lines of try_peek()? It'd be cool if it returned some structure that allowed fine grained control of the peeking:

```rust
let max_ranges = 32;
let max_bytes = 1024 * 1024 * 32;
let mut current_bytes = 0;
let mut ranges = Vec::new();
let mut peek = decoder.peek()
loop {
match peek.next() {
PeekResult::Range(range) => {
ranges.push(range);
current_bytes += range.end - range.start;
if ranges.len() > max_ranges { break }
if current_bytes > max_bytes { break }
PeekResult::End { break }
}
}
```

Here is another potential API from the original ticket:
```rust

// Create a decoder for decoding parquet data as above
let mut decoder: ParquetDecoderBuilder = ...;

// As the decoder up from what data it will need, start prefetching data if desired
while let Some(pre_request) = decoder.peek_next_requests() {
// note that this is a peek and if we call peek again in the
// future, we may get a different set of pre_requests (for example
// if the decoder has applied a row filter and ruled out
// some row groups or data pages)
start_prefetch(pre_request);
}
```

// push data to the decoder as before, but hopefully the reader
// will have already prefetched some of the data
**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the initial PushDecoder implementation in PR 7997 and the parent design discussion in issue 8000. Compare the proposed try_peek and peek_next_requests interfaces, then define an API that supports bounded range or byte peeking for prefetching; done means the design is agreed and the decoder can communicate upcoming data requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.