aws / aws/sagemaker-python-sdk

Local mode fails when referencing FinalMetricDataList

未关闭
#5,201 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
component: local mode type: bug
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

**Describe the bug**
When using local mode, referencing FinalMetricDataList leads to an error.

**To reproduce**

To create and run the pipeline:

```python
import os
import boto3
import sagemaker
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.fail_step import FailStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.pipeline_context import LocalPipelineSession
from sagemaker.workflow.steps import TrainingStep
from sagemaker.pytorch.estimator import PyTorch

execution_role = "arn:aws:iam::123456789012:role/MyPlaceholderRole"
boto_session = boto3.session.Session(region_name="eu-central-1")

localstack_hostname = os.getenv("LOCALSTACK_HOSTNAME")
localstack_edge_port = os.getenv("LOCALSTACK_EDGE_PORT")

sagemaker_session = LocalPipelineSession(
boto_session=boto_session,
default_bucket="some-bucket-name",
s3_endpoint_url=f"http://{localstack_hostname}:{localstack_edge_port}",
)

train_model_step = TrainingStep(
name="training_step",
estimator=PyTorch(
sagemaker_session=sagemaker_session,
py_version="py310",
framework_version="2.2",
role=execution_role,
instance_count=1,
entrypoint=["python3", "/opt/ml/code/main.py"],
instance_type="ml.c5.xlarge",
entry_point="train.py",
source_dir="src/steps/train_model_step",
metric_definitions=[
{
"Name": "ganloss",
"Regex": "GAN_loss=(0.138318);",
},
],
),
)
fail_step = FailStep(
name="FailPipeline",
error_message="Model score did not meet the required threshold.",
)
condition_step = ConditionStep(
name="CheckMetrics",
conditions=[
ConditionGreaterThanOrEqualTo(
left=train_model_step.properties.FinalMetricDataList[
"ganloss"
].Value,
right=0,
)
],
depends_on=[train_model_step],
if_steps=[],
else_steps=[fail_step],
)
pipeline = Pipeline(
name="my_pipeline",
steps=[train_model_step, condition_step],
sagemaker_session=sagemaker_session,
)
pipeline.upsert(
role_arn=execution_role,
)
execution = pipeline.start()
```

This is the file that is executed by model training:
src/steps/train_model_step/train.py

```python
print(
"GAN_loss=0.138318; Scaled_reg=2.654134; disc:[-0.017371,0.102429] real 93.3% gen 0.0% disc-combined=0.000000; disc_train_loss=1.374587; Loss = 16.020744; Iteration 0 took 0.704s; Elapsed=0s"
)
```

Furthermore, a localstack container needs to be running:

```sh
docker run -it -e SERVICES=s3,sts,kms -p4566:4566 -p 4571:4571 localstack/localstack:latest
```

**Expected behavior**
I expect the pipeline to run successfully in local mode without errors. This is only possible when running the pipeline on SageMaker without local mode.

**Screenshots or logs**

```
[06/10/25 11:02:32] INFO ===== Job Complete ===== image.py:325
INFO Pipeline step 'training_step' SUCCEEDED. entities.py:798
INFO Starting pipeline step: 'CheckMetrics' entities.py:807
INFO Pipeline step 'CheckMetrics' FAILED. Failure message is: {'Get': "Steps.training_step.FinalMetricDataList['ganloss'].Value"} is undefined. entities.py:802
INFO Pipeline execution 24084683-6ab9-4da1-a4b2-842e818630bd FAILED because step 'CheckMetrics' failed.
```

**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.243.3
- **Python version**: 3.12

贡献指南

打开贡献指南

调研方向

首先使用提供的 Python 示例、LocalPipelineSession 和 localstack 容器复现该 pipeline。检查本地模式下对 train_model_step.properties.FinalMetricDataList 的处理,以及所引用的训练入口点 src/steps/train_model_step/train.py。当 CheckMetrics 条件能够评估打印出的 ganloss 指标,并且 pipeline 在本地模式下成功完成时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python, pytorch
领域
machine-learning
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。