huggingface / huggingface/datasets
IterableDataset.to_dict() and to_polars() return an empty generator instead of the data
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
`IterableDataset.to_dict()` and `IterableDataset.to_polars()` each have a bare `yield` in their `batched=True` branch, which makes the whole method a generator function. Calling either one the way its docstring shows returns a generator that yields nothing, and the value the `else` branch returns is swallowed into `StopIteration.value`. Both are annotated `-> Union[dict, Iterator[dict]]` / `-> Union["pl.DataFrame", Iterator["pl.DataFrame"]]`, and the plain call is the only example in each docstring.
The sibling `to_pandas()` had the same shape and was changed to a returned generator expression in #8068; `to_dict` and `to_polars` were left as they were. `to_list()` is unaffected.
### Steps to reproduce the bug
```python
from datasets import Dataset
ds = Dataset.from_dict({"col": [0, 1, 2]}).to_iterable_dataset()
print(type(ds.to_dict())) #
print(list(ds.to_dict())) # []
ds.to_dict()["col"] # TypeError: 'generator' object is not subscriptable
```
The data is only reachable through the exception:
```python
g = ds.to_dict()
try:
next(g)
except StopIteration as e:
print(e.value) # {'col': [0, 1, 2]}
```
`ds.to_polars()` behaves identically. On the same object, `ds.to_pandas()` returns a DataFrame and `ds.to_list()` returns the rows.
### Expected behavior
`ds.to_dict()` returns `{'col': [0, 1, 2]}` and `ds.to_polars()` returns a `polars.DataFrame`, matching `Dataset.to_dict()` / `Dataset.to_polars()` and the non-iterator half of each return annotation. `batched=True` should keep yielding one batch at a time.
### Environment info
- datasets 5.0.2.dev0, main at `b7cb10b0e`
- python 3.11.15, Linux
- pyarrow 25.0.0, polars 1.43.2, pandas 3.0.5
Contributor guide
Research direction
Start at the IterableDataset.to_dict() and IterableDataset.to_polars() methods, then compare their batched branches with to_pandas() and the change in #8068. Verify the documented plain calls return the collected dictionary and Polars DataFrame, while batched=True still yields batches; reproduce the examples and run the relevant dataset tests if available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100