aws / aws/sagemaker-pytorch-training-toolkit
[bug] Torch does not find GPU on pytorch-training:1.10.0-gpu-py38 container
- Dominant language
- Python
- Stars
- 202
- Forks
- 96
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Torch does not find Cuda on GPU instance and official SageMaker training container
**To reproduce**
```
sudo docker pull 763104351884.dkr.ecr.eu-west-2.amazonaws.com/pytorch-training:1.10.0-gpu-py38-cu113-ubuntu20.04-sagemaker
sudo docker run -it --entrypoint /bin/bash 709fa9395949
python -c "import torch; print(torch.cuda.is_available()) -> False"
```
**Expected behavior**
`python -c "import torch; print(torch.cuda.is_available()) -> True`
**System information**
This command was run on SageMaker Notebook instance ml.p3.2xlarge (docker pull from console) and EC2 instance p3.2xlarge
Contributor guide
Research direction
Start by reproducing the reported Docker commands on the specified SageMaker or EC2 GPU instance and verify the container image and runtime details. Inspect the toolkit and the referenced SageMaker container build sources for GPU setup; done means torch.cuda.is_available() returns True in the pytorch-training:1.10.0-gpu-py38-cu113-ubuntu20.04-sagemaker container.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python, pytorch
- Domain
- cloud, devops, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100