aws / aws/sagemaker-python-sdk
Local mode fails when referencing FinalMetricDataList
- 主要语言
- Python
- 星标
- 2.3k
- 派生
- 1.3k
- 平均合并
- 1 天 22 小时
- 30 天内合并 PR
- 35
描述
**Describe the bug**
When using local mode, referencing FinalMetricDataList leads to an error.
**To reproduce**
To create and run the pipeline:
```python
import os
import boto3
import sagemaker
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.fail_step import FailStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.pipeline_context import LocalPipelineSession
from sagemaker.workflow.steps import TrainingStep
from sagemaker.pytorch.estimator import PyTorch
execution_role = "arn:aws:iam::123456789012:role/MyPlaceholderRole"
boto_session = boto3.session.Session(region_name="eu-central-1")
localstack_hostname = os.getenv("LOCALSTACK_HOSTNAME")
localstack_edge_port = os.getenv("LOCALSTACK_EDGE_PORT")
sagemaker_session = LocalPipelineSession(
boto_session=boto_session,
default_bucket="some-bucket-name",
s3_endpoint_url=f"http://{localstack_hostname}:{localstack_edge_port}",
)
train_model_step = TrainingStep(
name="training_step",
estimator=PyTorch(
sagemaker_session=sagemaker_session,
py_version="py310",
framework_version="2.2",
role=execution_role,
instance_count=1,
entrypoint=["python3", "/opt/ml/code/main.py"],
instance_type="ml.c5.xlarge",
entry_point="train.py",
source_dir="src/steps/train_model_step",
metric_definitions=[
{
"Name": "ganloss",
"Regex": "GAN_loss=(0.138318);",
},
],
),
)
fail_step = FailStep(
name="FailPipeline",
error_message="Model score did not meet the required threshold.",
)
condition_step = ConditionStep(
name="CheckMetrics",
conditions=[
ConditionGreaterThanOrEqualTo(
left=train_model_step.properties.FinalMetricDataList[
"ganloss"
].Value,
right=0,
)
],
depends_on=[train_model_step],
if_steps=[],
else_steps=[fail_step],
)
pipeline = Pipeline(
name="my_pipeline",
steps=[train_model_step, condition_step],
sagemaker_session=sagemaker_session,
)
pipeline.upsert(
role_arn=execution_role,
)
execution = pipeline.start()
```
This is the file that is executed by model training:
src/steps/train_model_step/train.py
```python
print(
"GAN_loss=0.138318; Scaled_reg=2.654134; disc:[-0.017371,0.102429] real 93.3% gen 0.0% disc-combined=0.000000; disc_train_loss=1.374587; Loss = 16.020744; Iteration 0 took 0.704s; Elapsed=0s"
)
```
Furthermore, a localstack container needs to be running:
```sh
docker run -it -e SERVICES=s3,sts,kms -p4566:4566 -p 4571:4571 localstack/localstack:latest
```
**Expected behavior**
I expect the pipeline to run successfully in local mode without errors. This is only possible when running the pipeline on SageMaker without local mode.
**Screenshots or logs**
```
[06/10/25 11:02:32] INFO ===== Job Complete ===== image.py:325
INFO Pipeline step 'training_step' SUCCEEDED. entities.py:798
INFO Starting pipeline step: 'CheckMetrics' entities.py:807
INFO Pipeline step 'CheckMetrics' FAILED. Failure message is: {'Get': "Steps.training_step.FinalMetricDataList['ganloss'].Value"} is undefined. entities.py:802
INFO Pipeline execution 24084683-6ab9-4da1-a4b2-842e818630bd FAILED because step 'CheckMetrics' failed.
```
**System information**
A description of your system. Please provide:
- **SageMaker Python SDK version**: 2.243.3
- **Python version**: 3.12
贡献指南
调研方向
首先使用提供的 Python 示例、LocalPipelineSession 和 localstack 容器复现该 pipeline。检查本地模式下对 train_model_step.properties.FinalMetricDataList 的处理,以及所引用的训练入口点 src/steps/train_model_step/train.py。当 CheckMetrics 条件能够评估打印出的 ganloss 指标,并且 pipeline 在本地模式下成功完成时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- aws, python, pytorch
- 领域
- machine-learning
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100