aws / aws/sagemaker-huggingface-inference-toolkit

Sagemaker endpoint inference Fails when following a tutorial

Open
#122 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Hello,
I have been trying to use a Llava endpoint in sagemaker when following this tutorial (https://github.com/aws-samples/multimodal-rag-on-slide-decks/blob/main/Blog1-TitanEmbeddings-LVM/notebooks/0_deploy_llava.ipynb).
I havent used the Cloudformation, I am just interested in deploying the model itself.
The endpoint is deployed however, when I try to invoke it i get the following error:

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "InternalServerException",
"message": "GET was unable to find an engine to execute this computation"
}

Not quite sure what to do and what the exact issue is.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the deployment and endpoint invocation in the 0_deploy_llava.ipynb tutorial, without the CloudFormation setup. Inspect the SageMaker inference configuration and the endpoint logs around the reported engine error. Done means identifying the configuration or compatibility cause and documenting or verifying a working model invocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.