aws / aws/sagemaker-pytorch-training-toolkit

[bug] Torch does not find GPU on pytorch-training:1.10.0-gpu-py38 container

Open
#232 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
202
Forks
96
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Torch does not find Cuda on GPU instance and official SageMaker training container

**To reproduce**
```
sudo docker pull 763104351884.dkr.ecr.eu-west-2.amazonaws.com/pytorch-training:1.10.0-gpu-py38-cu113-ubuntu20.04-sagemaker
sudo docker run -it --entrypoint /bin/bash 709fa9395949
python -c "import torch; print(torch.cuda.is_available()) -> False"
```

**Expected behavior**
`python -c "import torch; print(torch.cuda.is_available()) -> True`

**System information**
This command was run on SageMaker Notebook instance ml.p3.2xlarge (docker pull from console) and EC2 instance p3.2xlarge

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported Docker commands on the specified SageMaker or EC2 GPU instance and verify the container image and runtime details. Inspect the toolkit and the referenced SageMaker container build sources for GPU setup; done means torch.cuda.is_available() returns True in the pytorch-training:1.10.0-gpu-py38-cu113-ubuntu20.04-sagemaker container.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, python, pytorch
Domain
cloud, devops, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.