aws / aws/sagemaker-training-toolkit
SageMaker training toolkit reorders hyperparameters
- Dominant language
- Python
- Stars
- 530
- Forks
- 140
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 2
Description
Running a ScriptMode job on SageMaker modifies the order of the provided arguments.
For example, say my script `train.py` takes two arguments `--config` and `--batch_size`. If the `hyperparameter` argument on the `Estimator` class is set to `{'config': exp_config_path, 'batch_size': 10}`, then I would expect SageMaker training toolkit to invoke the script as follows:
```python train.py --config exp_config_path --batch_size 10```
However, the toolkit sorts the hyperparameters alphanumerically first before invoking the script, hence resulting in the following invocation:
```python train.py --batch_size 10 --config exp_config_path```
This happens because of a single line:
https://github.com/aws/sagemaker-training-toolkit/blob/628166c157751ae2a46fddc11a7a8cac765fb22c/src/sagemaker_training/mapping.py#L78
The issue with this reordering is because sometimes the order of the arguments matters. For example, when using [`jsonargparse`](https://jsonargparse.readthedocs.io/en/stable/) the order of invocations can result in two different argument settings. If the order is config>batch_size in script invocation, then `batch_size` argument is first loaded from the config file and subsequently is overridden by the command line argument `--batch_size 10`. On the other hand, if the order is batch_size>config, then the `batch_size` argument value is taken from the config.
I don't really see a reason why the hyperparameters should be sorted, so I think it would be safe to remove this sorting. This would be easy to fix by removing sorting in https://github.com/aws/sagemaker-training-toolkit/blob/628166c157751ae2a46fddc11a7a8cac765fb22c/src/sagemaker_training/mapping.py#L78?plain=1
## Reproducing the bug
Here's a sketch to reproduce the bug:
```python
from sagemaker.pytorch import PyTorch
# Setup the AWS role and session as usual
estimator = PyTorch(role=role,
entry_point='train.py',
instance_count=1,
instance_type="local",
sagemaker_session=local_sess,
output_path=output_path,
hyperparameters={'config': 'configs/config.yaml', 'batch_size': 10,},
)
estimator.fit()
```
Running this code will result in the invocation with reordered hyperparameters, which can be seen from the logs:
```
l12v045866-algo-1-09ky7 | Invoking script with the following command:
l12v045866-algo-1-09ky7 |
l12v045866-algo-1-09ky7 | /root/miniconda3/envs/conda_env/bin/python3.10 train.py --batch_size 10 --config configs/config.yaml
```
Instead I would expect the training toolkit to invoke the following command:
```
/root/miniconda3/envs/conda_env/bin/python3.10 train.py --config configs/config.yaml --batch_size 10
```
Contributor guide
Research direction
Start in src/sagemaker_training/mapping.py at the sorting operation referenced around line 78, then use the provided ScriptMode reproduction to observe the generated command. Done means hyperparameters are passed to train.py in the provided order, including the config-before-batch_size case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, python
- Domain
- devtools, machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100