meta-pytorch / meta-pytorch/data

MPRS blocks indefinitely

Open
#1,109 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🐛 Describe the bug
  1. Create a pipeline and use MPRS with 'spawn' (same behavior might potentially also be replicated with 'fork')
  2. Have the worker process fail to start due to an error during module import (other scenarios might potentially also provoke this)
  3. Main process now blocks endlessly in [x for x in dataloader]
  4. It's only possible to terminate the process by sending an KeyboardInterupt.

Solution: Add a timeout parameter to the MPRS constructor and make

File "torchdata/dataloader2/communication/protocol.py", line 104, in get_new_request
    response = self.request_queue.get(block=block)

fail after that timeout period.

Versions

https://github.com/pytorch/data/commit/e78ab6c9ec94f05f0a350ced7fe571f6863c20ec

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with torchdata/dataloader2/communication/protocol.py, especially get_new_request, and trace how the MPRS constructor creates or uses the request queue. Reproduce the failing pipeline with a worker that errors during module import and iteration over the dataloader. Done means the main process no longer blocks indefinitely and the request wait ends after the configured timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.