apache / apache/arrow-rs

add an example of parquet read in pipeline

Open
#9,320 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

next_row_group async api is powerful that allows us to overlap io and decoding. I felt like we should add a read_pipeline in the example.

```
async fn read_pipelined(data: Bytes) -> Result> {
let reader = InMemoryReader::new(data);
let builder = ParquetRecordBatchStreamBuilder::new(reader).await?;
let mut stream = builder.with_batch_size(8192).build()?;

// Channel buffers row group readers ahead of decode
// Buffer size of 2 means we can have up to 2 row groups fetched and waiting
let (tx, mut rx) = mpsc::channel(2);

// Spawn I/O task - continuously fetches row groups
let io_handle = tokio::spawn(async move {
while let Ok(Some(reader)) = stream.next_row_group().await {
// Send reader to decode task; if receiver is dropped, stop fetching
if tx.send(reader).await.is_err() {
break;
}
}
});

// Decode task - processes readers as they arrive
let mut batches = vec![];
while let Some(reader) = rx.recv().await {
// Decode in a blocking task to avoid blocking the async runtime
// This is important because decoding is CPU-bound
let decoded = tokio::task::spawn_blocking(move || {
reader.into_iter().collect::, _>>()
})
.await
.expect("decode task panicked")?;

batches.extend(decoded);
}

// Wait for I/O task to complete
io_handle.await.expect("I/O task panicked");

Ok(batches)
}

```

Contributor guide

Open the contributing guide

Research direction

The issue does not name a file, test, or example entry point. Start by locating existing Parquet read examples and the next_row_group API in the repository, then compare their structure with the supplied read_pipelined example. Done means the example demonstrates overlapping row-group I/O and decoding and builds with the existing examples.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.