apache / apache/arrow-rs

Unable to create `ParquetRecordBatchStreamBuilder` due to runtime issues

Open
#8,231 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Describe the bug**
Attempting to create a `ParquetRecordBatchStreamBuilder`, reading from S3 panics with the following error: `there is no reactor running, must be called from the context of a Tokio 1.x runtime`. The confusion for me is that the code is running within a Tokio runtime, v1.47.

**To Reproduce**
```rust
#[tokio::main]
async fn main() {
let path = "path/to/key";
let s3_store: Arc = Arc::new(
AmazonS3Builder::from_env()
.with_bucket_name(BUCKET_NAME)
.build()?,
);

let path = Path::from(key);
let reader = ParquetObjectReader::new(s3_store, path);
let builder = ParquetRecordBatchStreamBuilder::new(reader).await?;
}
```
The code panics on the last line, and a backtrace leads me to [this function](https://github.com/apache/arrow-rs/blob/1dacecba8e11cac307eea5d1a0f10c22d7f4a8b7/parquet/src/arrow/async_reader/store.rs#L197), which has a comment above it referencing something about async/IO.

**Expected behavior**
I'd expect the code above to *just work*^tm - am I doing something wrong here?

**Additional context**
There's a decent likelihood that I'm just misconfiguring something, but I figured I'd post this here either way so that someone else (or me, in the future) has the troubleshooting that will follow as a reference.

Contributor guide

Open the contributing guide

Research direction

Start with the ParquetRecordBatchStreamBuilder::new entry point and the async reader function in parquet/src/arrow/async_reader/store.rs around line 197. Run the provided Tokio and AmazonS3 reproduction to determine why the runtime is not recognized. Done means the builder can read from S3 without panicking under the shown Tokio runtime.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.