Horovod PyTorch Multi-GPU Only Using 1 GPU (Multi-Node Too)

For the Horovod + PyTorch + MNIST example, I created my own algorithm deriving this method. From what I can tell, it seems like SageMaker is only taking advantage of 1 GPU per node even when using ml.p3.16xlarge instances. hvd.size() will always return 1 and hvd.local_rank() always returns 0.

It probably has something to do with the `horovodrun` command under the hood. Take a look at the very bottom of [this page](https://horovod.readthedocs.io/en/stable/pytorch.html).

It requires extra params for launching multi GPU.

The reason I think this is the problem is if you look at [SageMaker Tensorflow Estimator's documentation](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator), there is an arg called `distributions`. According to [this article by AWS](https://aws.amazon.com/blogs/machine-learning/launching-tensorflow-distributed-training-easily-with-horovod-or-parameter-servers-in-amazon-sagemaker/) this is where you need to define your horovod (mpi) options. More specifically `hvd_processes_per_host`.

In contrast, the [SageMaker PyTorch Estimator's documentation](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html) has no such argument. Neither do their base classes ([Framework Estimator](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html#sagemaker.estimator.Framework) and so on...).

I could very well be doing something wrong, but literally all I did was change a few hyperparameters and use a larger dataset than MNIST (ImageNet).

Another interesting side effect is that after 25-33% of the first epoch, GPU utilization drops from ~86% all the way to 0%, however the GPU memory usage remains the same.

And finally the data partitioning doesn't seem to work, like I said hvd.local_rank() returns 0 and hvd.size() returns 1 so the dataset is read fully as 100% for each node even when each node has 8 GPUs.

GLOO & NCCL, PyTorch 1.4.0 & 1.5.0 are the same story.

**Here are some logs:** (note end batches take much much longer than the previous)
[1xNode 1xGPU Pastebin](https://pastebin.com/gj5Py9tv)

[1xNode 1xGPU **w/ forkserver** Pastebin](https://pastebin.com/0EwruPGP)
- Note: Seems like the GPU tanking is [solved by using fork](https://github.com/horovod/horovod/blob/a14f3cfc598bdb7d8bb38199ac04d715eb056c50/examples/pytorch_mnist.py#L126)...

[1xNode 4xGPU **w/ forkserver** Pastebin](https://pastebin.com/rayda9Jp) -- Interestingly enough, there was about a 30% speedup from 1GPU to 4GPU. Same instances just different GPU counts. However, again, `hvd.size()` reports 1.

[2xNode 1xGPU **w/ forkserver** Pastebin](https://pastebin.com/QwQkDXit)

Start with the Horovod examples/pytorch_mnist.py entry point and the SageMaker PyTorch Estimator and Framework Estimator documentation referenced in the report. Reproduce the issue with the listed single- and multi-node GPU configurations, checking horovodrun, hvd.size(), hvd.local_rank(), and dataset partitioning. Done means all GPUs and nodes are launched and reported correctly, with partitioned data.

Written by the indexing model from the issue text.