meta-pytorch / meta-pytorch/data
early stop worker got Exception Error
Open
Nobody has claimed this yet.
datapipes
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
dp = ...
dp = dp.sharding_filter()
rs = MultiProcessingReadingService(num_workers=4)
dataloader = DataLoader2(dp, reading_service=rs)
for _ in dataloader:
pass
dataloader.shutdown()
Exception: Can not expect any response without submitted request
This exception is thrown by __iter__ of _IterateQueueDataPipes(datapipes=[QueueWrapper, QueueWrapper, QueueWrapper, QueueWrapper])
related to https://github.com/pytorch/data/commit/901b48312179865657b90792aa03f0775ec50971
Versions
[pip3] numpy==1.26.4
[pip3] pytorch-wpe==0.0.1
[pip3] rotary-embedding-torch==0.6.4
[pip3] torch==2.1.1
[pip3] torch-complex==0.4.4
[pip3] torchaudio==2.1.1
[pip3] torchdata==0.7.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the example with DataLoader2, MultiProcessingReadingService, sharding_filter, and four workers, then trace the iter path of _IterateQueueDataPipes and the related commit. Done means the example can finish iteration and call dataloader.shutdown() without the “Can not expect any response without submitted request” exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100