apache / apache/arrow

[Python] Add `pyarrow.dataset.dataset` doesn't accept `RecordBatchReader`

Open
#40,264 0 comments 4 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The documentation suggests that `pyarrow.dataset.dataset` should accept a `RecordBatchReader` as a data source, however, this fails.

```python
import pyarrow
import pyarrow.dataset

batch = pyarrow.RecordBatch.from_pydict({'a': [1, 2, 3]})
reader = pyarrow.RecordBatchReader.from_batches(batch.schema, [batch])
dataset = pyarrow.dataset.dataset(reader)
```
```
Traceback (most recent call last):
File "", line 1, in
File "/.venv/lib/python3.11/site-packages/pyarrow/dataset.py", line 802, in dataset
raise TypeError(
TypeError: Expected a path-like, list of path-likes or a list of Datasets instead of the given type: RecordBatchReader
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.