apache / apache/arrow

[Python][dataset] Add support for python generator

Open
#39,923 2 comments 3 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Hi,

The python dataset module contains some checkings about the type of the source. I am sorry for my ignorance but I think a generator is also an Iterable (doc says it accepts Iterable) so when passing a generator to the source it should be checked against (GeneratorType) which is missing [here](https://github.com/apache/arrow/blob/main/python/pyarrow/dataset.py#L781-L800).

`dataset.py`
```python

if _is_path_like(source):
return _filesystem_dataset(source, **kwargs)
elif isinstance(source, (tuple, list)):
if all(_is_path_like(elem) for elem in source):
return _filesystem_dataset(source, **kwargs)
elif all(isinstance(elem, Dataset) for elem in source):
return _union_dataset(source, **kwargs)
elif all(isinstance(elem, (pa.RecordBatch, pa.Table))
for elem in source):
return _in_memory_dataset(source, **kwargs)
else:
unique_types = set(type(elem).__name__ for elem in source)
type_names = ', '.join('{}'.format(t) for t in unique_types)
raise TypeError(
'Expected a list of path-like or dataset objects, or a list '
'of batches or tables. The given list contains the following '
'types: {}'.format(type_names)
)
elif isinstance(source, (pa.RecordBatch, pa.Table)):
return _in_memory_dataset(source, **kwargs)

```

I wanted to pass a generator to the dataset.dataset(source=data_generator, schema=data_schema) but it gives me this error:

`TypeError: Expected a path-like, list of path-likes or a list of Datasets instead of the given type: generator`.

Is it possible to add a check for GeneratorType or is there any reason why generators are not supported as source in the dataset api ?

env:

- python 3.10.13
- pyarrow: 15.0.0

Thank you,
BR,
Ismail

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.