aws / aws/amazon-sagemaker-feedback
Categorical tuning hyperparameter with variable number of values determined during the ML pipeline execution start
- Dominant language
- No language data
- Stars
- 10
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
### Product Version
- [ ] Amazon SageMaker Studio Classic
- [ ] Amazon SageMaker Studio
- [x] It is not related to SageMaker Studio
### Product Category
Pipelines
### Description
I'd like to be able to set a different number of possible values for the categorical tuning hyperparameter during pipeline startup.
Possible solutions:
1. An additional parameter for CategoricalParameter to limit the number of values during runtime.
```python
val1 = ParameterString("Val1", default_value="Val1")
val2 = ParameterString("Val2", default_value="Val2")
val3 = ParameterString("Val3", default_value="Val3")
num_of_values = ParameterInteger("NumOfValues", default_value=2)
hparam_space= CategoricalParameter(values=[val1, val2, val3], num_of_values=num_of_values)
pipelines_params = [num_of_values , val1, val2, val3]
```
2. Removing duplicates from the list during runtime. This way, the user could limit the number of values by entering the same value multiple times.
```python
val1 = ParameterString("Val1", default_value="Val1")
val2 = ParameterString("Val2", default_value="Val2")
val3 = ParameterString("Val3", default_value="Val2")
hparam_space= CategoricalParameter(values=[val1, val2, val3])
pipelines_params = [val1, val2, val3]
```
3. A special value ignored by the tuner. For example, if <> were in the list, it would be ignored.
```python
val1 = ParameterString("Val1", default_value="Val1")
val2 = ParameterString("Val2", default_value="Val2")
val3 = ParameterString("Val3", default_value="<>")
hparam_space= CategoricalParameter(values=[val1, val2, val3])
pipelines_params = [val1, val2, val3]
```
4. A pipeline parameter of type list. In this case, such a parameter could be passed directly to the CategoricalParameter constructor. Such a pipeline parameter could also prove useful in other places.
```python
hparam_values= ParameterList("HParamValues", default_value=["Val1", "Val2"])
hparam_space= CategoricalParameter(values=hparam_values)
pipelines_params = [hparam_values]
```
5. Workflow function Split(value: Union[str, PipelineVariable], separator: Union[str, PipelineVariable]) being complementary to workflow function Join.
```python
hparam_values= ParameterString("HParamValues", default_value="Val1,Val2")
hparam_space= CategoricalParameter(values=Split(value=hparam_values, separator=","))
pipelines_params = [hparam_values]
```
Each of the above solutions should work in the context of the code below, running the pipeline with tuning step firing 2 jobs, once with the value for hparam: val1 and once with the value val2.
```python
from sagemaker.pytorch import PyTorch
from sagemaker.tuner import HyperparameterTuner
from sagemaker.workflow.steps import TuningStep
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import (
PipelineSession
)
if __name__ == "__main__":
pipeline_session = PipelineSession()
estimator = PyTorch(
sagemaker_session=pipeline_session,
instance_type='ml.m5.large',
instance_count=1,
framework_version="2.3",
py_version="py311",
source_dir='source',
entry_point='main.py',
metric_definitions=[
{'Name': 'valid:loss', 'Regex': 'valid_loss=([0-9]+\\.?[0-9]*)'}
]
)
<>
tuner = HyperparameterTuner(
estimator=estimator,
objective_metric_name='valid:loss',
objective_type='Minimize',
hyperparameter_ranges={
"hparam": hparam_space
},
max_jobs=2,
max_parallel_jobs=1,
base_tuning_job_name='test-tuning',
strategy='Grid',
metric_definitions=estimator.metric_definitions,
)
tuning_step = TuningStep(
name="Tuning",
step_args=tuner.fit(),
)
pipeline = Pipeline(
name="TestTuningPipeline",
parameters=pipelines_params,
steps=[tuning_step],
sagemaker_session=pipeline_session
)
pipeline.upsert()
execution = pipeline.start(
execution_display_name="TuningTest",
)
print(execution)
```
Sample **source/main.py**
```python
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--hparam", type=str, required=True)
args, _ = parser.parse_known_args()
print(f"Hparam: {args.hparam}")
print("valid_loss=0.1")
```
### Other Details
_No response_
Contributor guide
Research direction
The issue's entry points are CategoricalParameter, ParameterString/ParameterInteger, pipeline parameters, and TuningStep in the supplied Python example; start by tracing how these values are represented and passed when the pipeline starts. Compare the five proposed approaches and define what supported behavior should produce exactly two tuning jobs, then verify it with the shown main.py example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100