meta-pytorch / meta-pytorch/data
MPRS blocks indefinitely
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
- Create a pipeline and use MPRS with 'spawn' (same behavior might potentially also be replicated with 'fork')
- Have the worker process fail to start due to an error during module import (other scenarios might potentially also provoke this)
- Main process now blocks endlessly in
[x for x in dataloader] - It's only possible to terminate the process by sending an KeyboardInterupt.
Solution: Add a timeout parameter to the MPRS constructor and make
File "torchdata/dataloader2/communication/protocol.py", line 104, in get_new_request
response = self.request_queue.get(block=block)
fail after that timeout period.
Versions
https://github.com/pytorch/data/commit/e78ab6c9ec94f05f0a350ced7fe571f6863c20ec
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with torchdata/dataloader2/communication/protocol.py, especially get_new_request, and trace how the MPRS constructor creates or uses the request queue. Reproduce the failing pipeline with a worker that errors during module import and iteration over the dataloader. Done means the main process no longer blocks indefinitely and the request wait ends after the configured timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100