aws / aws/amazon-sagemaker-examples
[Example Request] Example to host llama 13b on sagemaker using neurons
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
https://github.com/aws/amazon-sagemaker-examples-community/blob/main/torchserve/inf2/llama2/llama-2-13b.ipynb
Could create an endpoint as above for llama 13b base, but it gives a timeout error on container primary for 13b chat.
For above, created the neuron artifacts for the 13b chat model using this -
https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md?plain=1#L56
Could start torchserve and run inference via curl command here, so the model artifacts look okay. But the same artifacts won't work in the first notebook reference link.
Contributor guide
Research direction
Start with the referenced torchserve/inf2/llama2/llama-2-13b.ipynb and compare it with the PyTorch Serve Inferentia2 Llama 2 README at the cited section. Reproduce the timeout for the 13B chat model, then determine why its neuron artifacts work with TorchServe and curl but not in the SageMaker endpoint. Done means the example hosts the chat model and supports successful inference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100