Megvii-BaseDetection / Megvii-BaseDetection/YOLOX

AttributeError: module 'torch._C' has no attribute '_cuda_setDevice

Open
#393 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
10.6k
Forks
2.5k
PR merge metrics
No merged PRs in 30d

Description

This is my training commands:
python tools/train.py -f exps/example/yolox_voc/yolox_voc_s.py --fp16 -o -c weights/yolox_s.pth.tar
training log:
2021-08-04 14:30:09.288 | INFO | yolox.core.trainer:before_train:125 - args: Namespace(batch_size=64, ckpt='weights/yolox_s.pth.tar', devices=None, dist_backend='nccl', dist_url=None, exp_file='exps/example/yolox_voc/yolox_voc_s.py', experiment_name='yolox_voc_s', fp16=False, local_rank=0, machine_rank=0, name=None, num_machines=1, occupy=False, opts=[], resume=False, start_epoch=None)
2021-08-04 14:30:09.291 | INFO | yolox.core.trainer:before_train:126 - exp value:
╒══════════════════╤════════════════════════════╕
│ keys │ values │
╞══════════════════╪════════════════════════════╡
│ seed │ None │
├──────────────────┼────────────────────────────┤
│ output_dir │ './YOLOX_outputs' │
├──────────────────┼────────────────────────────┤
│ print_interval │ 10 │
├──────────────────┼────────────────────────────┤
│ eval_interval │ 10 │
├──────────────────┼────────────────────────────┤
│ num_classes │ 8 │
├──────────────────┼────────────────────────────┤
│ depth │ 0.33 │
├──────────────────┼────────────────────────────┤
│ width │ 0.5 │
├──────────────────┼────────────────────────────┤
│ data_num_workers │ 4 │
├──────────────────┼────────────────────────────┤
│ input_size │ (640, 640) │
├──────────────────┼────────────────────────────┤
│ random_size │ (14, 26) │
├──────────────────┼────────────────────────────┤
│ data_dir │ None │
├──────────────────┼────────────────────────────┤
│ train_ann │ 'instances_train2017.json' │
├──────────────────┼────────────────────────────┤
│ val_ann │ 'instances_val2017.json' │
├──────────────────┼────────────────────────────┤
│ degrees │ 10.0 │
├──────────────────┼────────────────────────────┤
│ translate │ 0.1 │
├──────────────────┼────────────────────────────┤
│ scale │ (0.1, 2) │
├──────────────────┼────────────────────────────┤
│ mscale │ (0.8, 1.6) │
├──────────────────┼────────────────────────────┤
│ shear │ 2.0 │
├──────────────────┼────────────────────────────┤
│ perspective │ 0.0 │
├──────────────────┼────────────────────────────┤
│ enable_mixup │ True │
├──────────────────┼────────────────────────────┤
│ warmup_epochs │ 5 │
├──────────────────┼────────────────────────────┤
│ max_epoch │ 10 │
├──────────────────┼────────────────────────────┤
│ warmup_lr │ 0 │
├──────────────────┼────────────────────────────┤
│ basic_lr_per_img │ 0.00015625 │
├──────────────────┼────────────────────────────┤
│ scheduler │ 'yoloxwarmcos' │
├──────────────────┼────────────────────────────┤
│ no_aug_epochs │ 15 │
├──────────────────┼────────────────────────────┤
│ min_lr_ratio │ 0.05 │
├──────────────────┼────────────────────────────┤
│ ema │ True │
├──────────────────┼────────────────────────────┤
│ weight_decay │ 0.0005 │
├──────────────────┼────────────────────────────┤
│ momentum │ 0.9 │
├──────────────────┼────────────────────────────┤
│ exp_name │ 'yolox_voc_s' │
├──────────────────┼────────────────────────────┤
│ test_size │ (640, 640) │
├──────────────────┼────────────────────────────┤
│ test_conf │ 0.01 │
├──────────────────┼────────────────────────────┤
│ nmsthre │ 0.65 │
╘══════════════════╧════════════════════════════╛
2021-08-04 14:30:09.295 | ERROR | yolox.core.launch:launch:90 - An error has been caught in function 'launch', process 'MainProcess' (14416), thread 'MainThread' (23260):
Traceback (most recent call last):

File "tools/train.py", line 125, in
args=(exp, args),
│ └ Namespace(batch_size=64, ckpt='weights/yolox_s.pth.tar', devices=None, dist_backend='nccl', dist_url=None, exp_file='exps/exa...
└ ╒══════════════════╤════════════════════════════╕
│ keys │ values │
╞══════════════════╪═════...

File "d:\mmdetection\yolox-main\yolox\core\launch.py", line 90, in launch
main_func(*args)
│ └ (╒══════════════════╤════════════════════════════╕
│ │ keys │ values │
│ ╞══════════════════╪════...
└ <function main at 0x0000019D0818A558>

File "tools/train.py", line 104, in main
trainer.train()
│ └ <function Trainer.train at 0x0000019D093823A8>
└ <yolox.core.trainer.Trainer object at 0x0000019D0A907E48>

File "d:\mmdetection\yolox-main\yolox\core\trainer.py", line 69, in train
self.before_train()
│ └ <function Trainer.before_train at 0x0000019D0A8AD8B8>
└ <yolox.core.trainer.Trainer object at 0x0000019D0A907E48>

File "d:\mmdetection\yolox-main\yolox\core\trainer.py", line 129, in before_train
torch.cuda.set_device(self.local_rank)
│ │ │ │ └ 0
│ │ │ └ <yolox.core.trainer.Trainer object at 0x0000019D0A907E48>
│ │ └ <function set_device at 0x0000019D78EFCCA8>
│ └ <module 'torch.cuda' from 'D:\Anaconda\envs\yolox\lib\site-packages\torch\cuda\init.py'>
└ <module 'torch' from 'D:\Anaconda\envs\yolox\lib\site-packages\torch\init.py'>

File "D:\Anaconda\envs\yolox\lib\site-packages\torch\cuda_init_.py", line 264, in set_device
torch._C._cuda_setDevice(device)
│ │ └ 0
│ └ <module 'torch._C' from 'D:\Anaconda\envs\yolox\lib\site-packages\torch\_C.cp37-win_amd64.pyd'>
└ <module 'torch' from 'D:\Anaconda\envs\yolox\lib\site-packages\torch\init.py'>

AttributeError: module 'torch._C' has no attribute '_cuda_setDevice'
I don't know how to solve it.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tools/train.py and yolox/core/trainer.py, especially the torch.cuda.set_device call shown in the traceback. Check the PyTorch installation and GPU support in the reported Windows environment, then rerun the training command. Done means training proceeds past device initialization without the _cuda_setDevice error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.