zhanghang1989 / zhanghang1989/PyTorch-Encoding

File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init torch._C._cuda_init() RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (torchenc) mona@goku:~$ echo $CUDA_VISIBLE_DEVICES

Open
#368 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2k
Forks
448
PR merge metrics
No merged PRs in 30d

Description

Could you please suggest how I should fix this problem?

(torchenc) mona@goku:~$ python test_torch_encoding.py 
/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
Traceback (most recent call last):
  File "test_torch_encoding.py", line 5, in <module>
    model = encoding.models.get_model('DeepLab_ResNeSt269_PContext', pretrained=True).cuda()
  File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 463, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 381, in _apply
    param_applied = fn(param)
  File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 463, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/home/mona/venv/torchenc/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.
(torchenc) mona@goku:~$ echo $CUDA_VISIBLE_DEVICES

(torchenc) mona@goku:~$ nvidia-smi
Sat Jan 30 00:44:47 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 165...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   47C    P8     3W /  N/A |   1695MiB /  3911MiB |      7%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1155      G   /usr/lib/xorg/Xorg                133MiB |
|    0   N/A  N/A      1773      G   /usr/lib/xorg/Xorg               1124MiB |
|    0   N/A  N/A      1952      G   /usr/bin/gnome-shell              130MiB |
|    0   N/A  N/A      2329      G   ...gAAAAAAAAA --shared-files      167MiB |
|    0   N/A  N/A      2735      G   /usr/lib/firefox/firefox            3MiB |
|    0   N/A  N/A      2923      G   /usr/lib/firefox/firefox            1MiB |
|    0   N/A  N/A      3204      G   /usr/lib/firefox/firefox            1MiB |
|    0   N/A  N/A      3324      G   /usr/lib/firefox/firefox            1MiB |
|    0   N/A  N/A      3456      G   /usr/lib/firefox/firefox            1MiB |
|    0   N/A  N/A      3501      G   /usr/lib/firefox/firefox            1MiB |
|    0   N/A  N/A      3572      G   /usr/lib/firefox/firefox            1MiB |
|    0   N/A  N/A      3623      G   /usr/lib/firefox/firefox            3MiB |
|    0   N/A  N/A     21024      G   ...f_3837.log --shared-files       99MiB |
+-----------------------------------------------------------------------------+
(torchenc) mona@goku:~$ 

The code as in your Website:

import torch
import encoding

# Get the model
model = encoding.models.get_model('DeepLab_ResNeSt269_PContext', pretrained=True).cuda()
model.eval()

# Prepare the image
url = 'https://github.com/zhanghang1989/image-data/blob/master/' + \
      'encoding/segmentation/pcontext/2010_001829_org.jpg?raw=true'
filename = 'example.jpg'
img = encoding.utils.load_image(encoding.utils.download(url, filename)).cuda().unsqueeze(0)

# Make prediction
output = model.evaluate(img)
predict = torch.max(output, 1)[1].cpu().numpy() + 1

# Get color pallete for visualization
mask = encoding.utils.get_mask_pallete(predict, 'pascal_voc')
mask.save('output.png')

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running test_torch_encoding.py and reviewing the model's .cuda() call, the CUDA initialization warning, and the accompanying nvidia-smi output. Determine whether the failure is caused by the reported environment or by PyTorch-Encoding, then document a reproducible cause and confirmed resolution; the issue currently does not define a project change or test for completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.