aws / aws/amazon-sagemaker-feedback

Categorical tuning hyperparameter with variable number of values determined during the ML pipeline execution start

Open
#199 0 comments 3 reactions 0 assignees View on GitHub
feature-request
Dominant language
No language data
Stars
10
Forks
3
PR merge metrics
No merged PRs in 30d

Description

### Product Version

- [ ] Amazon SageMaker Studio Classic
- [ ] Amazon SageMaker Studio
- [x] It is not related to SageMaker Studio

### Product Category

Pipelines

### Description

I'd like to be able to set a different number of possible values for the categorical tuning hyperparameter during pipeline startup.

Possible solutions:

1. An additional parameter for CategoricalParameter to limit the number of values during runtime.

```python
val1 = ParameterString("Val1", default_value="Val1")
val2 = ParameterString("Val2", default_value="Val2")
val3 = ParameterString("Val3", default_value="Val3")
num_of_values = ParameterInteger("NumOfValues", default_value=2)
hparam_space= CategoricalParameter(values=[val1, val2, val3], num_of_values=num_of_values)
pipelines_params = [num_of_values , val1, val2, val3]
```

2. Removing duplicates from the list during runtime. This way, the user could limit the number of values by entering the same value multiple times.

```python
val1 = ParameterString("Val1", default_value="Val1")
val2 = ParameterString("Val2", default_value="Val2")
val3 = ParameterString("Val3", default_value="Val2")
hparam_space= CategoricalParameter(values=[val1, val2, val3])
pipelines_params = [val1, val2, val3]
```

3. A special value ignored by the tuner. For example, if <> were in the list, it would be ignored.

```python
val1 = ParameterString("Val1", default_value="Val1")
val2 = ParameterString("Val2", default_value="Val2")
val3 = ParameterString("Val3", default_value="<>")
hparam_space= CategoricalParameter(values=[val1, val2, val3])
pipelines_params = [val1, val2, val3]
```

4. A pipeline parameter of type list. In this case, such a parameter could be passed directly to the CategoricalParameter constructor. Such a pipeline parameter could also prove useful in other places.

```python
hparam_values= ParameterList("HParamValues", default_value=["Val1", "Val2"])
hparam_space= CategoricalParameter(values=hparam_values)
pipelines_params = [hparam_values]
```

5. Workflow function Split(value: Union[str, PipelineVariable], separator: Union[str, PipelineVariable]) being complementary to workflow function Join.

```python
hparam_values= ParameterString("HParamValues", default_value="Val1,Val2")
hparam_space= CategoricalParameter(values=Split(value=hparam_values, separator=","))
pipelines_params = [hparam_values]
```

Each of the above solutions should work in the context of the code below, running the pipeline with tuning step firing 2 jobs, once with the value for hparam: val1 and once with the value val2.
```python

from sagemaker.pytorch import PyTorch
from sagemaker.tuner import HyperparameterTuner
from sagemaker.workflow.steps import TuningStep
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import (
PipelineSession
)

if __name__ == "__main__":
pipeline_session = PipelineSession()

estimator = PyTorch(
sagemaker_session=pipeline_session,
instance_type='ml.m5.large',
instance_count=1,
framework_version="2.3",
py_version="py311",
source_dir='source',
entry_point='main.py',
metric_definitions=[
{'Name': 'valid:loss', 'Regex': 'valid_loss=([0-9]+\\.?[0-9]*)'}
]
)

<>

tuner = HyperparameterTuner(
estimator=estimator,
objective_metric_name='valid:loss',
objective_type='Minimize',
hyperparameter_ranges={
"hparam": hparam_space
},
max_jobs=2,
max_parallel_jobs=1,
base_tuning_job_name='test-tuning',
strategy='Grid',
metric_definitions=estimator.metric_definitions,
)

tuning_step = TuningStep(
name="Tuning",
step_args=tuner.fit(),
)

pipeline = Pipeline(
name="TestTuningPipeline",
parameters=pipelines_params,
steps=[tuning_step],
sagemaker_session=pipeline_session
)

pipeline.upsert()
execution = pipeline.start(
execution_display_name="TuningTest",
)
print(execution)
```

Sample **source/main.py**
```python
import argparse

if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--hparam", type=str, required=True)
args, _ = parser.parse_known_args()
print(f"Hparam: {args.hparam}")
print("valid_loss=0.1")
```

### Other Details

_No response_

Contributor guide

Open the contributing guide

Research direction

The issue's entry points are CategoricalParameter, ParameterString/ParameterInteger, pipeline parameters, and TuningStep in the supplied Python example; start by tracing how these values are represented and passed when the pipeline starts. Compare the five proposed approaches and define what supported behavior should produce exactly two tuning jobs, then verify it with the shown main.py example.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.