compatibility with pytorch DataLoader
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 231
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I was testing decord module with pytorch multiprocessed DataLoader.
When I use specific torch version and python version, importing decord before torch causes multiprocessed DataLoader to immediately freeze and doesn't proceed throughout the program.
(I tried to change the torch version and the problem remains, so I guess it can be narrowed down to specific python version.)
I made a small reproducing script for this issue.
import sys
import decord # when decord is imported before torch, the dataloader hangs
import torch
dataloader = torch.utils.data.DataLoader(dataset=range(100), num_workers=1)
print(sys.version) # 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0]
print(f"decord version: {decord.__version__}") # 0.6.0
print(f"torch version: {torch.__version__}") # 1.13.1+cu117
for batch in dataloader:
print(batch)
Anybody have an idea of why this kind of issue is happening?
Though I can fix this issue by simply switching the import order of decord and torch, but I am curious why this stuff happens at the first place.
Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the reproducing script from the issue with Python 3.10.9, decord 0.6.0, and torch 1.13.1+cu117, comparing the two import orders and the DataLoader with num_workers=1. Trace the import and worker-start paths; done means the reported hang is explained and the script completes regardless of import order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100