Expose a way to process a pa.RecordBatchFileReader without pa.dataset.Dataset
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
I am trying to apply an SQL query on a .feather file. This file cannot be materialized into RAM (too big) but each `pa.RecordBatch` can be.
I would like to be able to register the .feather file directly, or preferably via a `pa.ipc.RecordBatchFileReader`, then iter on the filtered `pa.RecordBatch` via `df.execute_stream()`.
I haven't found any way to do that right now that does not involve a `pa.dataset.Dataset`.
It is linked to the fact `pa.dataset.Dataset` cannot be instantiated from a `pa.ipc.RecordBatchFileReader`...
### Describe the solution you'd like
```python
import datafusion
import pyarrow.dataset as pdata
query = "SELECT * FROM archive WHERE foo = 'bar'"
handle = pa.memory_map(path, "rb")
reader = pa.ipc.RecordBatchFileReader(handle)
ctx = datafusion.SessionContext()
ctx.register_batch_reader("archive", reader)
df = ctx.sql(query)
for batch in df.execute_stream():
# do something
pass
```
### Describe alternatives you've considered
Right now, I am doing:
```python
import datafusion
import pyarrow.dataset as pdata
query = "SELECT * FROM archive WHERE foo = 'bar'"
dataset = pdata.dataset(path, format="ipc")
ctx = datafusion.SessionContext()
ctx.register_dataset("archive", dataset)
df = ctx.sql(query)
for batch in df.execute_stream():
# do something
pass
```
This works but it forces me to read the .feather file via `pa.dataset.Dataset`.
### Additional context
_No response_
Contributor guide
Research direction
Start with the Python-facing SessionContext.register_batch_reader and register_dataset entry points, then compare how the supplied pa.ipc.RecordBatchFileReader differs from the existing Dataset path. Done means a RecordBatchFileReader can be registered for the SQL query and consumed through df.execute_stream() without constructing a pa.dataset.Dataset; add focused coverage for the requested flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, sql
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100