meta-pytorch / meta-pytorch/data

`_DataPipeSerializationWrapper` doesn't work with multiprocessing Queue

Open
#934 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🐛 Describe the bug

After https://github.com/pytorch/data/pull/919 is landed, a hanging problem happens on MacOS or Windows, where spawn is used to create subprocesses by default. See: https://github.com/pytorch/data/actions/runs/3794926183
I was able to mitigate the issue by removing the SeializationWrapper from https://github.com/pytorch/data/blob/e15e1453967ce2f25f6fcd2838caadfd0e2fa811/torchdata/dataloader2/dataloader2.py#L112

And, the reason that the SerializationWrapper doesn't work is multiprocessing.Queue is attached to a DataPipe and sent to subprocesses. Even though I am able to solve my hanging problem in a different way, it's better to solve this problem directly via SerializationWrapper.

The following should be a minimum repro example

ctx = mp.get_context("spawn")
q = ctx.Queue()
dp = IterableWrapper(list(range(10)))
# Attach a Queue
dp.q = q
dl = DataLoader2(dp, reading_service=PrototypeMultiProcessingReadingService(2, "spawn"))
for d in dl:
    pass

cc: @NivekT

Versions

main

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with torchdata/dataloader2/dataloader2.py around line 112 and the referenced SerializationWrapper in torch/utils/data/datapipes/datapipe.py. Run the provided spawn-based multiprocessing Queue reproducer on macOS or Windows and inspect the linked PR 919 changes. Done means the wrapper no longer causes the DataLoader2 process to hang when a DataPipe has an attached Queue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.