add an example of parquet read in pipeline
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
next_row_group async api is powerful that allows us to overlap io and decoding. I felt like we should add a read_pipeline in the example.
```
async fn read_pipelined(data: Bytes) -> Result> {
let reader = InMemoryReader::new(data);
let builder = ParquetRecordBatchStreamBuilder::new(reader).await?;
let mut stream = builder.with_batch_size(8192).build()?;
// Channel buffers row group readers ahead of decode
// Buffer size of 2 means we can have up to 2 row groups fetched and waiting
let (tx, mut rx) = mpsc::channel(2);
// Spawn I/O task - continuously fetches row groups
let io_handle = tokio::spawn(async move {
while let Ok(Some(reader)) = stream.next_row_group().await {
// Send reader to decode task; if receiver is dropped, stop fetching
if tx.send(reader).await.is_err() {
break;
}
}
});
// Decode task - processes readers as they arrive
let mut batches = vec![];
while let Some(reader) = rx.recv().await {
// Decode in a blocking task to avoid blocking the async runtime
// This is important because decoding is CPU-bound
let decoded = tokio::task::spawn_blocking(move || {
reader.into_iter().collect::, _>>()
})
.await
.expect("decode task panicked")?;
batches.extend(decoded);
}
// Wait for I/O task to complete
io_handle.await.expect("I/O task panicked");
Ok(batches)
}
```
Contributor guide
Research direction
The issue does not name a file, test, or example entry point. Start by locating existing Parquet read examples and the next_row_group API in the repository, then compare their structure with the supplied read_pipelined example. Done means the example demonstrates overlapping row-group I/O and decoding and builds with the existing examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100