dmlc / dmlc/decord

compatibility with pytorch DataLoader

Open
#251 0 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
2.5k
Forks
231
PR merge metrics
No merged PRs in 30d

Description

Hi, I was testing decord module with pytorch multiprocessed DataLoader.
When I use specific torch version and python version, importing decord before torch causes multiprocessed DataLoader to immediately freeze and doesn't proceed throughout the program.
(I tried to change the torch version and the problem remains, so I guess it can be narrowed down to specific python version.)

I made a small reproducing script for this issue.

import sys
import decord  # when decord is imported before torch, the dataloader hangs
import torch

dataloader = torch.utils.data.DataLoader(dataset=range(100), num_workers=1)

print(sys.version)  # 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0]
print(f"decord version: {decord.__version__}")  # 0.6.0
print(f"torch version: {torch.__version__}")  # 1.13.1+cu117

for batch in dataloader:
    print(batch)

Anybody have an idea of why this kind of issue is happening?
Though I can fix this issue by simply switching the import order of decord and torch, but I am curious why this stuff happens at the first place.
Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reproducing script from the issue with Python 3.10.9, decord 0.6.0, and torch 1.13.1+cu117, comparing the two import orders and the DataLoader with num_workers=1. Trace the import and worker-start paths; done means the reported hang is explained and the script completes regardless of import order.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.