aws / aws/amazon-sagemaker-examples
Torch not compiled with CUDA enabled when deploying T5 using Triton
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/inference/nlp/realtime/triton/single-model/t5_pytorch_python-backend/t5_pytorch_python-backend.ipynb
**Describe the bug**
When following this notebook, getting an error when creating the endpoint. Endpoint creation fails with `error: creating server: Invalid argument - load failed for model '/opt/ml/model/::t5_pytorch': version 1 is at UNAVAILABLE state: Internal: AssertionError: `
error in the Cloudwatch.
**To reproduce**
Followed the above notebook for T5 model deployment, getting error at creating the endpoint.
**Logs**
`error: creating server: Invalid argument - load failed for model '/opt/ml/model/::t5_pytorch': version 1 is at UNAVAILABLE state: Internal: AssertionError: `
Contributor guide
Research direction
Start with the linked t5_pytorch_python-backend.ipynb and reproduce the T5 deployment, then inspect the CloudWatch logs for the full assertion behind the CUDA error. Compare the notebook's PyTorch and deployment configuration with the Triton endpoint requirements. Done means the endpoint is created successfully and the T5 model loads without the reported UNAVAILABLE error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100