Azure / Azure/MachineLearningNotebooks

CUDA issue on the Compute Instance

未關閉
#1,839 0 則留言 2 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Jupyter Notebook
星號
4.4k
分支
2.6k
PR 合併指標
30 天內沒有已合併 PR

描述

PyTorch cannot detect GPUs on the Compute Instance with NVIDIA A100 GPUs. `nvidia-smi` and `nvcc --version` can return the installed CUDA version and CUDA toolkit versions. But PyTorch returns the following:

```bash
# python -c "import torch; print(torch.cuda.is_available())"
/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py:80: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /opt/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:112.)
return torch._C._cuda_getDeviceCount() > 0
False
```

I tried to pull PyTorch container from Nvidia NGC, AML base images, or compile PyTorch using pip/conda with GPU support. All have the same CUDA issue. I also cannot compile NCCL test on the Compute Instance with GPUs.

However, everything works fine on the Computer Cluster VMs. NCCL compiles/runs file and PyTorch has no issue with CUDA. I suspect CUDA is installed but not configured correctly on the Compute Instance VMs.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

先在 Compute Instance 上重現 `python -c "import torch; print(torch.cuda.is_available())"`,並與正常運作的 Computer Cluster VMs 比較。檢查回報的 `nvidia-smi`、`nvcc --version` 輸出以及 CUDA 初始化錯誤,然後確認 PyTorch 能偵測到 A100 GPU,且 NCCL 測試能夠編譯並執行。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python, pytorch
領域
cloud, infrastructure, machine-learning
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
需要釐清
新手友好度
25/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。