apache / apache/arrow-rs

Add an API to provide the reason / what is being requested in IO

Open
#8,157 2 comments 1 reaction 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

So we basically have a 2nd file parser here 🤔 I'm wondering if instead of creating yet-another-parser -- even though it's partial -- we could ask the decoder to provide us a "reason" or a "trace" on the individual read requests. For example, we could extend `AsyncFileReader` with an implemented-by-default method:

```rust
trait AsyncFileReader {
// all the current methods stay!

fn get_bytes_with_trace(&mut self, range: Range, trace: Trace) -> BoxFuture<'_, parquet::errors::Result> {
// ignore trace by default
self.get_bytes(range)
}

// same for the other two methods...
}

// bikeshed whatever `Trace` is, maybe use :http::Extensions?
```

_Originally posted by @crepererum in https://github.com/apache/arrow-rs/pull/7971#discussion_r2262364717_

Contributor guide

Open the contributing guide

Research direction

Start with the AsyncFileReader trait and the discussion in PR #7971 referenced by the issue. Determine the agreed shape for a trace or reason and which read methods should accept it; done means the API design is resolved and the requested IO context can reach individual reads without another parser.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.