huggingface / huggingface/datasets

IterableDataset.to_dict() and to_polars() return an empty generator instead of the data

Open Beginner friendly
#8,381 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

`IterableDataset.to_dict()` and `IterableDataset.to_polars()` each have a bare `yield` in their `batched=True` branch, which makes the whole method a generator function. Calling either one the way its docstring shows returns a generator that yields nothing, and the value the `else` branch returns is swallowed into `StopIteration.value`. Both are annotated `-> Union[dict, Iterator[dict]]` / `-> Union["pl.DataFrame", Iterator["pl.DataFrame"]]`, and the plain call is the only example in each docstring.

The sibling `to_pandas()` had the same shape and was changed to a returned generator expression in #8068; `to_dict` and `to_polars` were left as they were. `to_list()` is unaffected.

### Steps to reproduce the bug

```python
from datasets import Dataset

ds = Dataset.from_dict({"col": [0, 1, 2]}).to_iterable_dataset()

print(type(ds.to_dict())) #
print(list(ds.to_dict())) # []
ds.to_dict()["col"] # TypeError: 'generator' object is not subscriptable
```

The data is only reachable through the exception:

```python
g = ds.to_dict()
try:
next(g)
except StopIteration as e:
print(e.value) # {'col': [0, 1, 2]}
```

`ds.to_polars()` behaves identically. On the same object, `ds.to_pandas()` returns a DataFrame and `ds.to_list()` returns the rows.

### Expected behavior

`ds.to_dict()` returns `{'col': [0, 1, 2]}` and `ds.to_polars()` returns a `polars.DataFrame`, matching `Dataset.to_dict()` / `Dataset.to_polars()` and the non-iterator half of each return annotation. `batched=True` should keep yielding one batch at a time.

### Environment info

- datasets 5.0.2.dev0, main at `b7cb10b0e`
- python 3.11.15, Linux
- pyarrow 25.0.0, polars 1.43.2, pandas 3.0.5

Contributor guide

Open the contributing guide

Research direction

Start at the IterableDataset.to_dict() and IterableDataset.to_polars() methods, then compare their batched branches with to_pandas() and the change in #8068. Verify the documented plain calls return the collected dictionary and Polars DataFrame, while batched=True still yields batches; reproduce the examples and run the relevant dataset tests if available.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.