aws / aws/sagemaker-python-sdk

Model builder unable to

未关闭
#5,667 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

PySDK Version
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

Describe the bug
ModelBuilder fails when building/deploying a model package whose BaseModel has HubContentName set but is missing HubContentVersion and/or RecipeName. These fields are returned as Unassigned from the DescribeModelPackage API response, and ModelBuilder does not handle this — it assumes all three
fields are populated and crashes during the build step.

This is a real problem for model packages created by automated pipelines (e.g., fine-tuning jobs that call CreateModelPackage) where the producer only sets HubContentName but not the version or recipe. The user has no way to patch these fields through the ModelBuilder API — the only workaround
is to manually resolve them before calling build().

To reproduce

python
from sagemaker.serve import ModelBuilder
from sagemaker.core.resources import ModelPackage

# A model package where BaseModel has HubContentName but missing HubContentVersion and RecipeName
mp_arn = "arn:aws:sagemaker:us-east-1:123456789012:model-package/my-fine-tuned-models/1"
mp = ModelPackage.get(model_package_name=mp_arn, region="us-east-1")

# Inspect — HubContentVersion and RecipeName are Unassigned
base_model = mp.inference_specification.containers[0].base_model
print(base_model.hub_content_name) # e.g. "huggingface-reasoning-qwen3-32b" ✅
print(base_model.hub_content_version) # Unassigned ❌
print(base_model.recipe_name) # Unassigned ❌

role = "arn:aws:iam::123456789012:role/SageMakerRole"
model_builder = ModelBuilder(model=mp, role_arn=role)
model_builder.build(model_name="my-deployment")
# ^^^ Fails — ModelBuilder cannot resolve the model without version and recipe

Workaround (manually patch before build):

python
from sagemaker.core.resources import HubContent
from sagemaker.core.utils.utils import Unassigned

base_model = mp.inference_specification.containers[0].base_model
if isinstance(base_model.hub_content_version, Unassigned):
hc = HubContent.get(
hub_content_type="Model",
hub_name="SageMakerPublicHub",
hub_content_name=base_model.hub_content_name,
)
base_model.hub_content_version = hc.hub_content_version
if isinstance(base_model.recipe_name, Unassigned) or not base_model.recipe_name:
base_model.recipe_name = "verl-grpo-rlaif-qwen-3-32b-lora"

# Now build() works
model_builder = ModelBuilder(model=mp, role_arn=role)
model_builder.build(model_name="my-deployment")
model_builder.deploy(endpoint_name="my-deployment", instance_type="ml.g5.48xlarge")

Expected behavior
ModelBuilder should gracefully handle model packages where HubContentVersion and/or RecipeName are Unassigned. Specifically:
- If HubContentName is present but HubContentVersion is missing, ModelBuilder should resolve it automatically by calling DescribeHubContent on SageMakerPublicHub (as the workaround does).
- If RecipeName is missing, ModelBuilder should either infer a default recipe from the hub content metadata or allow the user to pass it as a parameter.

Screenshots or logs
N/A — error occurs during model_builder.build() when it attempts to use the Unassigned values.

System information
- **SageMaker Python SDK version**: 3.6.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: DJL LMI (Large Model Inference)
- **Framework version**: DJL 0.36.0, LMI 20.0.0
- **Python version**: 3.12
- **CPU or GPU**: GPU (ml.g5.48xlarge — 8x NVIDIA A10G)
- **Custom Docker image (Y/N)**: N

Additional context
- The root cause is split: the model package producer (e.g., fine-tuning pipeline) should ideally populate all three BaseModel fields (HubContentName, HubContentVersion, RecipeName). But ModelBuilder should also be resilient to partially-populated model packages, since it has enough information (HubContentName) to resolve the missing fields itself.
- The workaround is functional but requires the user to know internal SDK types (Unassigned, HubContent.get) which are not documented.

贡献指南

打开贡献指南

调研方向

从通过 ModelBuilder.build() 进行 Python 复现开始,检查 ModelPackage 返回的 BaseModel 值为 Unassigned 时是如何处理的。将该流程与使用 HubContent.get 和 SageMakerPublicHub 的文档化 workaround 进行比较;当 build 能够在 HubContentVersion 和 RecipeName 缺失时不发生崩溃,并且该行为由适当的回归测试覆盖时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
cloud, machine-learning
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。