aws / aws/amazon-sagemaker-examples
hvd.size() gave 1 even if I use mutiple GPUs instance
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Hi,
I am running example [tensorflow_keras_CIFAR10.ipynb](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/keras_script_mode_pipe_mode_horovod/tensorflow_keras_CIFAR10.ipynb).
I set the distributions to enable mpi and use ml.p3.8xlarge instance.
But hvd.size() always return 1 for me.
Contributor guide
Research direction
Start with tensorflow_keras_CIFAR10.ipynb and inspect the SageMaker distribution settings for MPI and the ml.p3.8xlarge instance. Run the example while checking the Horovod size reported by the training process; done means the example documents or produces the expected multi-GPU world size instead of 1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, keras, tensorflow
- Domain
- cloud, distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100