aws / aws/sagemaker-python-sdk

Model builder unable to

Open
#5,667 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

PySDK Version
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

Describe the bug
ModelBuilder fails when building/deploying a model package whose BaseModel has HubContentName set but is missing HubContentVersion and/or RecipeName. These fields are returned as Unassigned from the DescribeModelPackage API response, and ModelBuilder does not handle this — it assumes all three
fields are populated and crashes during the build step.

This is a real problem for model packages created by automated pipelines (e.g., fine-tuning jobs that call CreateModelPackage) where the producer only sets HubContentName but not the version or recipe. The user has no way to patch these fields through the ModelBuilder API — the only workaround
is to manually resolve them before calling build().

To reproduce

python
from sagemaker.serve import ModelBuilder
from sagemaker.core.resources import ModelPackage

# A model package where BaseModel has HubContentName but missing HubContentVersion and RecipeName
mp_arn = "arn:aws:sagemaker:us-east-1:123456789012:model-package/my-fine-tuned-models/1"
mp = ModelPackage.get(model_package_name=mp_arn, region="us-east-1")

# Inspect — HubContentVersion and RecipeName are Unassigned
base_model = mp.inference_specification.containers[0].base_model
print(base_model.hub_content_name) # e.g. "huggingface-reasoning-qwen3-32b" ✅
print(base_model.hub_content_version) # Unassigned ❌
print(base_model.recipe_name) # Unassigned ❌

role = "arn:aws:iam::123456789012:role/SageMakerRole"
model_builder = ModelBuilder(model=mp, role_arn=role)
model_builder.build(model_name="my-deployment")
# ^^^ Fails — ModelBuilder cannot resolve the model without version and recipe

Workaround (manually patch before build):

python
from sagemaker.core.resources import HubContent
from sagemaker.core.utils.utils import Unassigned

base_model = mp.inference_specification.containers[0].base_model
if isinstance(base_model.hub_content_version, Unassigned):
hc = HubContent.get(
hub_content_type="Model",
hub_name="SageMakerPublicHub",
hub_content_name=base_model.hub_content_name,
)
base_model.hub_content_version = hc.hub_content_version
if isinstance(base_model.recipe_name, Unassigned) or not base_model.recipe_name:
base_model.recipe_name = "verl-grpo-rlaif-qwen-3-32b-lora"

# Now build() works
model_builder = ModelBuilder(model=mp, role_arn=role)
model_builder.build(model_name="my-deployment")
model_builder.deploy(endpoint_name="my-deployment", instance_type="ml.g5.48xlarge")

Expected behavior
ModelBuilder should gracefully handle model packages where HubContentVersion and/or RecipeName are Unassigned. Specifically:
- If HubContentName is present but HubContentVersion is missing, ModelBuilder should resolve it automatically by calling DescribeHubContent on SageMakerPublicHub (as the workaround does).
- If RecipeName is missing, ModelBuilder should either infer a default recipe from the hub content metadata or allow the user to pass it as a parameter.

Screenshots or logs
N/A — error occurs during model_builder.build() when it attempts to use the Unassigned values.

System information
- **SageMaker Python SDK version**: 3.6.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: DJL LMI (Large Model Inference)
- **Framework version**: DJL 0.36.0, LMI 20.0.0
- **Python version**: 3.12
- **CPU or GPU**: GPU (ml.g5.48xlarge — 8x NVIDIA A10G)
- **Custom Docker image (Y/N)**: N

Additional context
- The root cause is split: the model package producer (e.g., fine-tuning pipeline) should ideally populate all three BaseModel fields (HubContentName, HubContentVersion, RecipeName). But ModelBuilder should also be resilient to partially-populated model packages, since it has enough information (HubContentName) to resolve the missing fields itself.
- The workaround is functional but requires the user to know internal SDK types (Unassigned, HubContent.get) which are not documented.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.