meta-pytorch / meta-pytorch/data
[Nodes] ParallelMap state is not guraranteed with in_order=False
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
Currently ParallelMap does not guarantee reproducibility if in_order = False, we should add a .. warning to the docstring and the flag, and also think about ways to have some level of guarantee
https://github.com/pytorch/data/blob/main/torchdata/nodes/map.py#L285-L286
Versions
nightly
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading torchdata/nodes/map.py around lines 285-286, including the ParallelMap docstring and the in_order flag documentation. Document that state reproducibility is not guaranteed when in_order=False, then review the surrounding implementation to scope the separate question of what guarantees might be possible. Done means the warning is clear and the reproducibility limitation is accurately described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100