Buffered shuffle on odps dataset
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 257
- PR merge metrics
- No merged PRs in 30d
Description
https://discuss.pytorch.org/t/how-to-shuffle-an-iterable-dataset/64130/5
- torch 1.8.1
[torch.utils.data.BufferedShuffleDataset](https://pytorch.org/docs/1.8.1/data.html#torch.utils.data.BufferedShuffleDataset)
- latest
[torch.utils.data.datapipes.iter.combinatorics.ShuffleIterDataPipe](https://github.com/pytorch/pytorch/blob/615dd2508846d03bc1f4e21898cc2694535011fe/torch/utils/data/datapipes/iter/combinatorics.py#L55)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked PyTorch BufferedShuffleDataset documentation and the referenced ShuffleIterDataPipe implementation, then compare them with EasyNLP's ODPS dataset entry point. The issue does not name a repository file or test, so identify the affected dataset path and define completion criteria before changing code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100