aws / aws/amazon-sagemaker-examples

hvd.size() gave 1 even if I use mutiple GPUs instance

Open
#1,821 2 comments 0 reactions 0 assignees View on GitHub
bug:distributed-training status: awaiting response
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Hi,

I am running example [tensorflow_keras_CIFAR10.ipynb](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/keras_script_mode_pipe_mode_horovod/tensorflow_keras_CIFAR10.ipynb).
I set the distributions to enable mpi and use ml.p3.8xlarge instance.
But hvd.size() always return 1 for me.

Contributor guide

Open the contributing guide

Research direction

Start with tensorflow_keras_CIFAR10.ipynb and inspect the SageMaker distribution settings for MPI and the ml.p3.8xlarge instance. Run the example while checking the Horovod size reported by the training process; done means the example documents or produces the expected multi-GPU world size instead of 1.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, keras, tensorflow
Domain
cloud, distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.