aws / aws/amazon-sagemaker-feedback
Error cause for incorrect configuration of HyperparameterTuner in ML pipeline if the incorrect configuration is a result of pipeline parameters.
- Dominant language
- No language data
- Stars
- 10
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
### Product Version
- [ ] Amazon SageMaker Studio Classic
- [ ] Amazon SageMaker Studio
- [x] It is not related to SageMaker Studio
### Product Category
Pipelines
### Description
If the TuningJob configuration fails due to pipeline parameters, the tuning step terminates almost immediately without starting any training jobs. The only information I could find about the reason for the failure is: "No Tuning Job Configuration found." As shown in the image below:
The below code illustrate the issue:
```python
from sagemaker.parameter import ParameterRange
from sagemaker.pytorch import PyTorch
from sagemaker.tuner import HyperparameterTuner, CategoricalParameter, ContinuousParameter
from sagemaker.workflow.parameters import ParameterFloat
from sagemaker.workflow.steps import TuningStep
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import (
PipelineSession
)
if __name__ == "__main__":
pipeline_session = PipelineSession()
estimator = PyTorch(
sagemaker_session=pipeline_session,
instance_type='ml.m5.large',
instance_count=1,
framework_version="2.3",
py_version="py311",
source_dir='source',
entry_point='main.py',
metric_definitions=[
{'Name': 'valid:loss', 'Regex': 'valid_loss=([0-9]+\\.?[0-9]*)'}
]
)
hparam_min = ParameterFloat("hparam_min", default_value=1.0)
hparam_max = ParameterFloat("hparam_max", default_value=0.0)
tuner = HyperparameterTuner(
estimator=estimator,
objective_metric_name='valid:loss',
objective_type='Minimize',
hyperparameter_ranges={
"hparam": ContinuousParameter(hparam_min, hparam_max)
},
max_jobs=2,
max_parallel_jobs=1,
base_tuning_job_name='test-tuning',
strategy='Random',
metric_definitions=estimator.metric_definitions,
)
tuning_step = TuningStep(
name="Tuning",
step_args=tuner.fit(),
)
pipeline = Pipeline(
name="TestTuningPipeline",
parameters=[
hparam_min,
hparam_max
],
steps=[tuning_step],
sagemaker_session=pipeline_session
)
pipeline.upsert()
execution = pipeline.start(
execution_display_name="TuningTest",
)
print(execution)
```
The pipeline with the tuning step will be created correctly and started, but it will fail almost immediately because the ContinuousParameter has a larger min_value than max_value. The problem is that I can't find any information about the cause of the error, and in a larger program, finding the cause can be problematic.
### Other Details
_No response_
Contributor guide
Research direction
Start with the supplied Python reproducer, focusing on HyperparameterTuner, ContinuousParameter, TuningStep, and pipeline.start(). Trace the tuning-step failure for the invalid min/max bounds and identify where the generic "No Tuning Job Configuration found" message is surfaced. Done means the pipeline exposes the configuration cause clearly when parameter values are invalid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100