apache / apache/datafusion

Expose a way to process a pa.RecordBatchFileReader without pa.dataset.Dataset

Open
#20,435 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I am trying to apply an SQL query on a .feather file. This file cannot be materialized into RAM (too big) but each `pa.RecordBatch` can be.
I would like to be able to register the .feather file directly, or preferably via a `pa.ipc.RecordBatchFileReader`, then iter on the filtered `pa.RecordBatch` via `df.execute_stream()`.

I haven't found any way to do that right now that does not involve a `pa.dataset.Dataset`.
It is linked to the fact `pa.dataset.Dataset` cannot be instantiated from a `pa.ipc.RecordBatchFileReader`...

### Describe the solution you'd like

```python
import datafusion
import pyarrow.dataset as pdata

query = "SELECT * FROM archive WHERE foo = 'bar'"

handle = pa.memory_map(path, "rb")
reader = pa.ipc.RecordBatchFileReader(handle)

ctx = datafusion.SessionContext()
ctx.register_batch_reader("archive", reader)
df = ctx.sql(query)

for batch in df.execute_stream():
# do something
pass
```

### Describe alternatives you've considered

Right now, I am doing:
```python
import datafusion
import pyarrow.dataset as pdata

query = "SELECT * FROM archive WHERE foo = 'bar'"
dataset = pdata.dataset(path, format="ipc")

ctx = datafusion.SessionContext()
ctx.register_dataset("archive", dataset)
df = ctx.sql(query)

for batch in df.execute_stream():
# do something
pass
```
This works but it forces me to read the .feather file via `pa.dataset.Dataset`.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Python-facing SessionContext.register_batch_reader and register_dataset entry points, then compare how the supplied pa.ipc.RecordBatchFileReader differs from the existing Dataset path. Done means a RecordBatchFileReader can be registered for the SQL query and consumed through df.execute_stream() without constructing a pa.dataset.Dataset; add focused coverage for the requested flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, sql
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.