aws / aws/sagemaker-python-sdk
Local mode fails when referencing FinalMetricDataList
- Langage dominant
- Python
- Étoiles
- 2.3k
- Forks
- 1.3k
- Merge moyen
- 1 j 22 h
- PR mergées (30 j)
- 35
Description
**Describe the bug**
When using local mode, referencing FinalMetricDataList leads to an error.
**To reproduce**
To create and run the pipeline:
```python
import os
import boto3
import sagemaker
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.fail_step import FailStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.pipeline_context import LocalPipelineSession
from sagemaker.workflow.steps import TrainingStep
from sagemaker.pytorch.estimator import PyTorch
execution_role = "arn:aws:iam::123456789012:role/MyPlaceholderRole"
boto_session = boto3.session.Session(region_name="eu-central-1")
localstack_hostname = os.getenv("LOCALSTACK_HOSTNAME")
localstack_edge_port = os.getenv("LOCALSTACK_EDGE_PORT")
sagemaker_session = LocalPipelineSession(
boto_session=boto_session,
default_bucket="some-bucket-name",
s3_endpoint_url=f"http://{localstack_hostname}:{localstack_edge_port}",
)
train_model_step = TrainingStep(
name="training_step",
estimator=PyTorch(
sagemaker_session=sagemaker_session,
py_version="py310",
framework_version="2.2",
role=execution_role,
instance_count=1,
entrypoint=["python3", "/opt/ml/code/main.py"],
instance_type="ml.c5.xlarge",
entry_point="train.py",
source_dir="src/steps/train_model_step",
metric_definitions=[
{
"Name": "ganloss",
"Regex": "GAN_loss=(0.138318);",
},
],
),
)
fail_step = FailStep(
name="FailPipeline",
error_message="Model score did not meet the required threshold.",
)
condition_step = ConditionStep(
name="CheckMetrics",
conditions=[
ConditionGreaterThanOrEqualTo(
left=train_model_step.properties.FinalMetricDataList[
"ganloss"
].Value,
right=0,
)
],
depends_on=[train_model_step],
if_steps=[],
else_steps=[fail_step],
)
pipeline = Pipeline(
name="my_pipeline",
steps=[train_model_step, condition_step],
sagemaker_session=sagemaker_session,
)
pipeline.upsert(
role_arn=execution_role,
)
execution = pipeline.start()
```
This is the file that is executed by model training:
src/steps/train_model_step/train.py
```python
print(
"GAN_loss=0.138318; Scaled_reg=2.654134; disc:[-0.017371,0.102429] real 93.3% gen 0.0% disc-combined=0.000000; disc_train_loss=1.374587; Loss = 16.020744; Iteration 0 took 0.704s; Elapsed=0s"
)
```
Furthermore, a localstack container needs to be running:
```sh
docker run -it -e SERVICES=s3,sts,kms -p4566:4566 -p 4571:4571 localstack/localstack:latest
```
**Expected behavior**
I expect the pipeline to run successfully in local mode without errors. This is only possible when running the pipeline on SageMaker without local mode.
**Screenshots or logs**
```
[06/10/25 11:02:32] INFO ===== Job Complete ===== image.py:325
INFO Pipeline step 'training_step' SUCCEEDED. entities.py:798
INFO Starting pipeline step: 'CheckMetrics' entities.py:807
INFO Pipeline step 'CheckMetrics' FAILED. Failure message is: {'Get': "Steps.training_step.FinalMetricDataList['ganloss'].Value"} is undefined. entities.py:802
INFO Pipeline execution 24084683-6ab9-4da1-a4b2-842e818630bd FAILED because step 'CheckMetrics' failed.
```
**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.243.3
- **Python version**: 3.12
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par reproduire le pipeline avec l’exemple Python fourni, LocalPipelineSession et le conteneur localstack. Inspectez la gestion en mode local de train_model_step.properties.FinalMetricDataList ainsi que le point d’entrée d’entraînement référencé, src/steps/train_model_step/train.py. Le travail est terminé lorsque la condition CheckMetrics évalue la métrique ganloss affichée et que le pipeline s’exécute jusqu’à son terme avec succès en mode local.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- aws, python, pytorch
- Domaine
- machine-learning
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 45/100