[Python][dataset] Add support for python generator
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Hi,
The python dataset module contains some checkings about the type of the source. I am sorry for my ignorance but I think a generator is also an Iterable (doc says it accepts Iterable) so when passing a generator to the source it should be checked against (GeneratorType) which is missing [here](https://github.com/apache/arrow/blob/main/python/pyarrow/dataset.py#L781-L800).
`dataset.py`
```python
if _is_path_like(source):
return _filesystem_dataset(source, **kwargs)
elif isinstance(source, (tuple, list)):
if all(_is_path_like(elem) for elem in source):
return _filesystem_dataset(source, **kwargs)
elif all(isinstance(elem, Dataset) for elem in source):
return _union_dataset(source, **kwargs)
elif all(isinstance(elem, (pa.RecordBatch, pa.Table))
for elem in source):
return _in_memory_dataset(source, **kwargs)
else:
unique_types = set(type(elem).__name__ for elem in source)
type_names = ', '.join('{}'.format(t) for t in unique_types)
raise TypeError(
'Expected a list of path-like or dataset objects, or a list '
'of batches or tables. The given list contains the following '
'types: {}'.format(type_names)
)
elif isinstance(source, (pa.RecordBatch, pa.Table)):
return _in_memory_dataset(source, **kwargs)
```
I wanted to pass a generator to the dataset.dataset(source=data_generator, schema=data_schema) but it gives me this error:
`TypeError: Expected a path-like, list of path-likes or a list of Datasets instead of the given type: generator`.
Is it possible to add a check for GeneratorType or is there any reason why generators are not supported as source in the dataset api ?
env:
- python 3.10.13
- pyarrow: 15.0.0
Thank you,
BR,
Ismail
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.