aws / aws/sagemaker-huggingface-inference-toolkit

Unable to deploy on SageMaker diffusers dropped `device_type="auto"`

Open
#139 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Based on the Python script below, we’re unable to deploy the model because `diffusers` ≥ 0.28 no longer accepts the device type **"auto"**. Consequently, the GPU isn’t detected when the endpoint starts.

```python
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel

iam = boto3.client("iam")
role = iam.get_role(RoleName="my-ml-sagemaker-role")["Role"]["Arn"]

# Hub model configuration – see https://huggingface.co/models
hub = {
"HF_MODEL_ID": "stable-diffusion-v1-5/stable-diffusion-v1-5",
"HF_TASK": "text-to-image",
}

# Create a Hugging Face model
huggingface_model = HuggingFaceModel(
transformers_version="4.49.0",
pytorch_version="2.6.0",
py_version="py312",
env=hub,
role=role,
)

# Deploy the model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1, # number of instances
instance_type="ml.g4dn.4xlarge", # EC2 instance type
)

image_bytes = predictor.predict({"inputs": "Astronaut riding a horse"})

# Display the generated image with PIL
import io
from PIL import Image
image = Image.open(io.BytesIO(image_bytes))
```

```bash
aws sagemaker-runtime invoke-endpoint \
--endpoint-name huggingface-pytorch-inference-2025-05-14-17-08-18-830 \
--body "fileb://input_file.txt" output_file.txt
```

This returns:

```
An error occurred (ModelError) when calling the InvokeEndpoint operation:
Received client error (400) from primary with message:
{
"code": 400,
"type": "InternalServerException",
"message": "auto not supported. Supported strategies are: balanced"
}
```

It looks like the SageMaker Hugging Face inference toolkit needs an update ? It use `balanced` only if there is more than 2 GPUs.

https://github.com/aws/sagemaker-huggingface-inference-toolkit/blob/main/src/sagemaker_huggingface_inference_toolkit/diffusers_utils.py#L43

This issue might be related, so I'm pinning it : https://github.com/huggingface/diffusers/issues/11555

Any guidance on how to resolve this SageMaker deployment issue would be greatly appreciated.
Thank you!

Contributor guide

Open the contributing guide

Research direction

Start with src/sagemaker_huggingface_inference_toolkit/diffusers_utils.py around line 43 and compare its device strategy with the diffusers version used in the reported SageMaker script. Reproduce the endpoint invocation and verify that deployment detects the GPU without the “auto not supported” error. Done means the deployment works with the current diffusers behavior and the supported strategy is handled correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.