aws / aws/sagemaker-training-toolkit

Pass args to training script entrypoint for MPI-based Distributed training

Open
#69 3 comments 0 reactions 0 assignees View on GitHub
type: enhancement
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

**Describe the feature you'd like**
Pass arguments to the training script while using Horovod via MPI for Distributed training.

**Current Situation**
~~Only~~ ProcessRunner supports passing hyperparameters
https://github.com/aws/sagemaker-training-toolkit/blob/c357433d6fdbc43a896b25bd126c46f689ddb73c/src/sagemaker_training/process.py#L105-L109

~~MPIRunner doesn't support it.~~
MPIRunner supports it: https://github.com/aws/sagemaker-training-toolkit/blob/c357433d6fdbc43a896b25bd126c46f689ddb73c/src/sagemaker_training/mpi.py#L41-L45

**How would this feature be used? Please describe.**
Example API would be
```
mpi_options = '-verbose -x orte_base_help_aggregate=0'
estimator = MXNet(
entry_point='hvd_resnet_mx.sh',
role=role,
train_instance_type='ml.p3.8xlarge',
train_instance_count=2,
image_name=image,
framework_version='1.6.0',
py_version='py3',
hyperparameters={'sagemaker_mpi_enabled': True,
'sagemaker_mpi_custom_mpi_options': mpi_options,
'sagemaker_mpi_num_of_processes_per_host': 4},
sagemaker_session=sagemaker_session)
```
Where entry-point script is
`hvd_resnet_mx.sh`

```
! pygmentize hvd_resnet_launcher.sh
./hvd_resnet_mx.py --num-epochs 5
```

**Describe alternatives you've considered**
~~Right now, one has to use ProcessRunner instead of MPIRunner to pass bash script for training~~

```
estimator = MXNet(
entry_point='hvd_resnet_launcher.sh',
role=role,
train_instance_type='ml.p3.8xlarge',
train_instance_count=2,
image_name=image,
framework_version='1.6.0',
py_version='py3',
hyperparameters={'sagemaker_parameter_server_enabled': True
},
sagemaker_session=sagemaker_session)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.