aws / aws/amazon-sagemaker-examples

Torch not compiled with CUDA enabled when deploying T5 using Triton

Open
#4,651 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/inference/nlp/realtime/triton/single-model/t5_pytorch_python-backend/t5_pytorch_python-backend.ipynb

**Describe the bug**
When following this notebook, getting an error when creating the endpoint. Endpoint creation fails with `error: creating server: Invalid argument - load failed for model '/opt/ml/model/::t5_pytorch': version 1 is at UNAVAILABLE state: Internal: AssertionError: `
error in the Cloudwatch.
**To reproduce**
Followed the above notebook for T5 model deployment, getting error at creating the endpoint.

**Logs**
`error: creating server: Invalid argument - load failed for model '/opt/ml/model/::t5_pytorch': version 1 is at UNAVAILABLE state: Internal: AssertionError: `

Contributor guide

Open the contributing guide

Research direction

Start with the linked t5_pytorch_python-backend.ipynb and reproduce the T5 deployment, then inspect the CloudWatch logs for the full assertion behind the CUDA error. Compare the notebook's PyTorch and deployment configuration with the Triton endpoint requirements. Done means the endpoint is created successfully and the T5 model loads without the reported UNAVAILABLE error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.