meta-pytorch / meta-pytorch/data
Invalidate `DataLoader2Iterator` would potentially violate other iterators by `finalize_iteration` reading service
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
We do finalize_iteration when __next__ is called for the invalid DataLoader2Iterator (let's call it Iter_A):
However, as a new DataLoader2Iterator (Iter_B) has been created and the same ReadingService has already called initialize here:
https://github.com/pytorch/data/blob/c42587a828d05f24f6f0586d17d3e9d55e1433ed/torchdata/dataloader2/dataloader2.py#L135
When someone called next(Iter_A) afterwards, it will finalize_iteration of the same ReadingService. Then, the Iter_B would exit earlier than expected.
Solution
We should move all logic from __next__ to __init__ for DataLoader2Iterator
cc: @NivekT
Versions
main branch
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in torchdata/dataloader2/dataloader2.py, especially the next logic at lines 67-68 and DataLoader2Iterator initialization around line 135. Trace how ReadingService.initialize and finalize_iteration interact across two iterators. Done means invalidating Iter_A no longer finalizes the service used by Iter_B, which should continue until its own iteration ends.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100