aws / aws/sagemaker-training-toolkit

Don't set `sagemaker_s3_output` via hyperparameter

Open
#104 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
530
Forks
140
Avg merge
1h 12m
Merged PRs (30d)
2

Description

**Describe the feature you'd like**
I discovered an undocumented feature where we can pass a hyperparameter named `sagemaker_s3_output` with an S3 URI. This will result in being able to store data in `opt/ml/output/intermediate` and that will get uploaded to the S3 URI during the training job.

I would like to take advantage of this, though I have 2 concerns.
1. hyperparameter should be reserved exclusively for actual hyperparameter, and not configuration information
2. Where this becomes a real issue is if we warm start a tuning job with different `sagemaker_s3_output`. `sagemaker_s3_output` would have an impact on the tuning strategy even though it shouldn't.

**How would this feature be used? Please describe.**
The `sagemaker_s3_output` value should be set via the [SageMaker CreateTrainingJob Environment parameter](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#sagemaker-CreateTrainingJob-request-Environment).

The main issue with this is that [SageMaker CreateHyperParameterTuningJob](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateHyperParameterTuningJob.html) has no Environment parameter (this is another feature request I've submitted to the SageMaker team).

**Describe alternatives you've considered**
We could just pass `sagemaker_s3_output` as a hyperparameter and just try to make sure they don't change between warm starts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.