Unable to create `ParquetRecordBatchStreamBuilder` due to runtime issues
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Describe the bug**
Attempting to create a `ParquetRecordBatchStreamBuilder`, reading from S3 panics with the following error: `there is no reactor running, must be called from the context of a Tokio 1.x runtime`. The confusion for me is that the code is running within a Tokio runtime, v1.47.
**To Reproduce**
```rust
#[tokio::main]
async fn main() {
let path = "path/to/key";
let s3_store: Arc = Arc::new(
AmazonS3Builder::from_env()
.with_bucket_name(BUCKET_NAME)
.build()?,
);
let path = Path::from(key);
let reader = ParquetObjectReader::new(s3_store, path);
let builder = ParquetRecordBatchStreamBuilder::new(reader).await?;
}
```
The code panics on the last line, and a backtrace leads me to [this function](https://github.com/apache/arrow-rs/blob/1dacecba8e11cac307eea5d1a0f10c22d7f4a8b7/parquet/src/arrow/async_reader/store.rs#L197), which has a comment above it referencing something about async/IO.
**Expected behavior**
I'd expect the code above to *just work*^tm - am I doing something wrong here?
**Additional context**
There's a decent likelihood that I'm just misconfiguring something, but I figured I'd post this here either way so that someone else (or me, in the future) has the troubleshooting that will follow as a reference.
Contributor guide
Research direction
Start with the ParquetRecordBatchStreamBuilder::new entry point and the async reader function in parquet/src/arrow/async_reader/store.rs around line 197. Run the provided Tokio and AmazonS3 reproduction to determine why the runtime is not recognized. Done means the builder can read from S3 without panicking under the shown Tokio runtime.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100