aws / aws/sagemaker-python-sdk

Model builder unable to

Offen
#5,667 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
2.3k
Forks
1.3k
Ø Merge
1 T. 22 Std.
Gemergte PRs (30 T.)
35

Beschreibung

PySDK Version
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

Describe the bug
ModelBuilder fails when building/deploying a model package whose BaseModel has HubContentName set but is missing HubContentVersion and/or RecipeName. These fields are returned as Unassigned from the DescribeModelPackage API response, and ModelBuilder does not handle this — it assumes all three
fields are populated and crashes during the build step.

This is a real problem for model packages created by automated pipelines (e.g., fine-tuning jobs that call CreateModelPackage) where the producer only sets HubContentName but not the version or recipe. The user has no way to patch these fields through the ModelBuilder API — the only workaround
is to manually resolve them before calling build().

To reproduce

python
from sagemaker.serve import ModelBuilder
from sagemaker.core.resources import ModelPackage

# A model package where BaseModel has HubContentName but missing HubContentVersion and RecipeName
mp_arn = "arn:aws:sagemaker:us-east-1:123456789012:model-package/my-fine-tuned-models/1"
mp = ModelPackage.get(model_package_name=mp_arn, region="us-east-1")

# Inspect — HubContentVersion and RecipeName are Unassigned
base_model = mp.inference_specification.containers[0].base_model
print(base_model.hub_content_name) # e.g. "huggingface-reasoning-qwen3-32b" ✅
print(base_model.hub_content_version) # Unassigned ❌
print(base_model.recipe_name) # Unassigned ❌

role = "arn:aws:iam::123456789012:role/SageMakerRole"
model_builder = ModelBuilder(model=mp, role_arn=role)
model_builder.build(model_name="my-deployment")
# ^^^ Fails — ModelBuilder cannot resolve the model without version and recipe

Workaround (manually patch before build):

python
from sagemaker.core.resources import HubContent
from sagemaker.core.utils.utils import Unassigned

base_model = mp.inference_specification.containers[0].base_model
if isinstance(base_model.hub_content_version, Unassigned):
hc = HubContent.get(
hub_content_type="Model",
hub_name="SageMakerPublicHub",
hub_content_name=base_model.hub_content_name,
)
base_model.hub_content_version = hc.hub_content_version
if isinstance(base_model.recipe_name, Unassigned) or not base_model.recipe_name:
base_model.recipe_name = "verl-grpo-rlaif-qwen-3-32b-lora"

# Now build() works
model_builder = ModelBuilder(model=mp, role_arn=role)
model_builder.build(model_name="my-deployment")
model_builder.deploy(endpoint_name="my-deployment", instance_type="ml.g5.48xlarge")

Expected behavior
ModelBuilder should gracefully handle model packages where HubContentVersion and/or RecipeName are Unassigned. Specifically:
- If HubContentName is present but HubContentVersion is missing, ModelBuilder should resolve it automatically by calling DescribeHubContent on SageMakerPublicHub (as the workaround does).
- If RecipeName is missing, ModelBuilder should either infer a default recipe from the hub content metadata or allow the user to pass it as a parameter.

Screenshots or logs
N/A — error occurs during model_builder.build() when it attempts to use the Unassigned values.

System information
- **SageMaker Python SDK version**: 3.6.0
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: DJL LMI (Large Model Inference)
- **Framework version**: DJL 0.36.0, LMI 20.0.0
- **Python version**: 3.12
- **CPU or GPU**: GPU (ml.g5.48xlarge — 8x NVIDIA A10G)
- **Custom Docker image (Y/N)**: N

Additional context
- The root cause is split: the model package producer (e.g., fine-tuning pipeline) should ideally populate all three BaseModel fields (HubContentName, HubContentVersion, RecipeName). But ModelBuilder should also be resilient to partially-populated model packages, since it has enough information (HubContentName) to resolve the missing fields itself.
- The workaround is functional but requires the user to know internal SDK types (Unassigned, HubContent.get) which are not documented.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne mit der Python-Reproduktion über ModelBuilder.build() und untersuche, wie von ModelPackage zurückgegebene BaseModel-Werte behandelt werden, wenn sie Unassigned sind. Vergleiche diesen Ablauf mit dem dokumentierten Workaround unter Verwendung von HubContent.get und SageMakerPublicHub; als erledigt gilt, dass build fehlendes HubContentVersion und RecipeName ohne Absturz verarbeitet und dieses Verhalten durch einen geeigneten Regressionstest abgedeckt ist.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, python
Bereich
cloud, machine-learning
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.