meta-pytorch / meta-pytorch/data
DataLoader2 with multiprocess raise exception: Can not request next item while we are still waiting response for previous request
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
dp = ...
dp = dp.sharding_filter()
rs = MultiProcessingReadingService(num_workers=4)
dataloader = DataLoader2(dp, reading_service=rs)
for _ in dataloader:
pass
dataloader.shutdown()
Exception: Can not request next item while we are still waiting response for previous request
This exception is thrown by __iter__ of _IterateQueueDataPipes(datapipes=[QueueWrapper, QueueWrapper, QueueWrapper, QueueWrapper])
Versions
[pip3] numpy==1.26.4
[pip3] onnx==1.15.0
[pip3] onnxconverter-common==1.13.0
[pip3] onnxruntime==1.15.1
[pip3] skl2onnx==1.16.0
[pip3] torch==2.2.1
[pip3] torchaudio==2.2.1
[pip3] torchdata==0.7.1
[pip3] torchvision==0.17.1
[pip3] triton==2.2.0
[conda] numpy 1.26.4 pypi_0 pypi
[conda] torch 2.2.1 pypi_0 pypi
[conda] torchaudio 2.2.1 pypi_0 pypi
[conda] torchvision 0.17.1 pypi_0 pypi
[conda] triton
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided DataLoader2 and MultiProcessingReadingService example with torchdata 0.7.1, then trace the _IterateQueueDataPipes path while iterating. Done means the loop completes without the pending-response exception and dataloader.shutdown() works normally.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100