huggingface / huggingface/datasets
interleave_datasets resets shuffle state
- Dominant language
- Python
- Stars
- 22k
- Forks
- 3.4k
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 17
Description
### Describe the bug
```
import datasets
import torch.utils.data
def gen(shards):
yield {"shards": shards}
def main():
dataset = datasets.IterableDataset.from_generator(
gen,
gen_kwargs={'shards': list(range(25))}
)
dataset = dataset.shuffle(buffer_size=1)
dataset = datasets.interleave_datasets(
[dataset, dataset], probabilities=[1, 0], stopping_strategy="all_exhausted"
)
dataloader = torch.utils.data.DataLoader(
dataset,
batch_size=8,
num_workers=8,
)
for i, batch in enumerate(dataloader):
print(batch)
if i >= 10:
break
if __name__ == "__main__":
main()
```
### Steps to reproduce the bug
Run the script, it will output
```
{'shards': [tensor([ 0, 8, 16, 24, 0, 8, 16, 24])]}
{'shards': [tensor([ 1, 9, 17, 1, 9, 17, 1, 9])]}
{'shards': [tensor([ 2, 10, 18, 2, 10, 18, 2, 10])]}
{'shards': [tensor([ 3, 11, 19, 3, 11, 19, 3, 11])]}
{'shards': [tensor([ 4, 12, 20, 4, 12, 20, 4, 12])]}
{'shards': [tensor([ 5, 13, 21, 5, 13, 21, 5, 13])]}
{'shards': [tensor([ 6, 14, 22, 6, 14, 22, 6, 14])]}
{'shards': [tensor([ 7, 15, 23, 7, 15, 23, 7, 15])]}
{'shards': [tensor([ 0, 8, 16, 24, 0, 8, 16, 24])]}
{'shards': [tensor([17, 1, 9, 17, 1, 9, 17, 1])]}
{'shards': [tensor([18, 2, 10, 18, 2, 10, 18, 2])]}
```
### Expected behavior
The shards should be shuffled.
### Environment info
- `datasets` version: 3.0.0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.25.0
- PyArrow version: 17.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0
Contributor guide
Assessment
This issue has not been assessed yet.