apache / apache/fluss-rust

[python] Add asyncio-native streaming API for bounded log reading

Open
#545 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
58
Forks
50
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss-rust/issues) and found nothing similar.

### Description

`LogScanner.to_arrow_batch_reader()` currently returns a synchronous `pyarrow.RecordBatchReader` that blocks the calling thread on each `__next__()` call. This might be acceptable for Arrow interop (e.g., feeding into DuckDB, Polars etc), but is not suitable for asyncio-native Python code.

So, ideally we should add an async counterpart, e.g., `async for batch in scanner.read_batches()` which yields `RecordBatch` objects without blocking the event loop.

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.