aws / aws/amazon-sagemaker-feedback

Error cause for incorrect configuration of HyperparameterTuner in ML pipeline if the incorrect configuration is a result of pipeline parameters.

Open
#196 0 comments 2 reactions 0 assignees View on GitHub
feature-request
Dominant language
No language data
Stars
10
Forks
3
PR merge metrics
No merged PRs in 30d

Description

### Product Version

- [ ] Amazon SageMaker Studio Classic
- [ ] Amazon SageMaker Studio
- [x] It is not related to SageMaker Studio

### Product Category

Pipelines

### Description

If the TuningJob configuration fails due to pipeline parameters, the tuning step terminates almost immediately without starting any training jobs. The only information I could find about the reason for the failure is: "No Tuning Job Configuration found." As shown in the image below:

Image

The below code illustrate the issue:
```python

from sagemaker.parameter import ParameterRange
from sagemaker.pytorch import PyTorch
from sagemaker.tuner import HyperparameterTuner, CategoricalParameter, ContinuousParameter
from sagemaker.workflow.parameters import ParameterFloat
from sagemaker.workflow.steps import TuningStep
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import (
PipelineSession
)

if __name__ == "__main__":
pipeline_session = PipelineSession()

estimator = PyTorch(
sagemaker_session=pipeline_session,
instance_type='ml.m5.large',
instance_count=1,
framework_version="2.3",
py_version="py311",
source_dir='source',
entry_point='main.py',
metric_definitions=[
{'Name': 'valid:loss', 'Regex': 'valid_loss=([0-9]+\\.?[0-9]*)'}
]
)

hparam_min = ParameterFloat("hparam_min", default_value=1.0)
hparam_max = ParameterFloat("hparam_max", default_value=0.0)

tuner = HyperparameterTuner(
estimator=estimator,
objective_metric_name='valid:loss',
objective_type='Minimize',
hyperparameter_ranges={
"hparam": ContinuousParameter(hparam_min, hparam_max)
},
max_jobs=2,
max_parallel_jobs=1,
base_tuning_job_name='test-tuning',
strategy='Random',
metric_definitions=estimator.metric_definitions,
)

tuning_step = TuningStep(
name="Tuning",
step_args=tuner.fit(),
)

pipeline = Pipeline(
name="TestTuningPipeline",
parameters=[
hparam_min,
hparam_max
],
steps=[tuning_step],
sagemaker_session=pipeline_session
)

pipeline.upsert()
execution = pipeline.start(
execution_display_name="TuningTest",
)
print(execution)
```
The pipeline with the tuning step will be created correctly and started, but it will fail almost immediately because the ContinuousParameter has a larger min_value than max_value. The problem is that I can't find any information about the cause of the error, and in a larger program, finding the cause can be problematic.

### Other Details

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Python reproducer, focusing on HyperparameterTuner, ContinuousParameter, TuningStep, and pipeline.start(). Trace the tuning-step failure for the invalid min/max bounds and identify where the generic "No Tuning Job Configuration found" message is surfaced. Done means the pipeline exposes the configuration cause clearly when parameter values are invalid.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.