huggingface / huggingface/datasets

interleave_datasets resets shuffle state

Open
#7,156 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
22k
Forks
3.4k
Avg merge
5d 7h
Merged PRs (30d)
17

Description

### Describe the bug

```
import datasets
import torch.utils.data

def gen(shards):
yield {"shards": shards}

def main():
dataset = datasets.IterableDataset.from_generator(
gen,
gen_kwargs={'shards': list(range(25))}
)
dataset = dataset.shuffle(buffer_size=1)
dataset = datasets.interleave_datasets(
[dataset, dataset], probabilities=[1, 0], stopping_strategy="all_exhausted"
)

dataloader = torch.utils.data.DataLoader(
dataset,
batch_size=8,
num_workers=8,
)

for i, batch in enumerate(dataloader):
print(batch)
if i >= 10:
break

if __name__ == "__main__":
main()
```

### Steps to reproduce the bug

Run the script, it will output

```
{'shards': [tensor([ 0, 8, 16, 24, 0, 8, 16, 24])]}
{'shards': [tensor([ 1, 9, 17, 1, 9, 17, 1, 9])]}
{'shards': [tensor([ 2, 10, 18, 2, 10, 18, 2, 10])]}
{'shards': [tensor([ 3, 11, 19, 3, 11, 19, 3, 11])]}
{'shards': [tensor([ 4, 12, 20, 4, 12, 20, 4, 12])]}
{'shards': [tensor([ 5, 13, 21, 5, 13, 21, 5, 13])]}
{'shards': [tensor([ 6, 14, 22, 6, 14, 22, 6, 14])]}
{'shards': [tensor([ 7, 15, 23, 7, 15, 23, 7, 15])]}
{'shards': [tensor([ 0, 8, 16, 24, 0, 8, 16, 24])]}
{'shards': [tensor([17, 1, 9, 17, 1, 9, 17, 1])]}
{'shards': [tensor([18, 2, 10, 18, 2, 10, 18, 2])]}
```

### Expected behavior

The shards should be shuffled.

### Environment info

- `datasets` version: 3.0.0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.25.0
- PyArrow version: 17.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.