aws / aws/amazon-sagemaker-examples
[Llama2 inferentia] : runtime error when invoking endpoint through boto3
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/aws-trainium-inferentia-finetuning-deployment/llama-2-trainium-inferentia-finetuning-deployment.ipynb
**Describe the bug**
Using a Lambda function with boto3 to query the neuron llama2 7b f model deployed on a ML INF2 XLARGE instance, the invoke endpoint operation fails with the following message:
```
{
"errorMessage": "An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message \"{\n \"code\": 400,\n \"type\": \"BadRequestException\",\n \"message\": \"Parameter model_name is required.\"\n}\n\". See https://us-east-2.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logEventViewer:group=/aws/sagemaker/Endpoints/testllamaneuron in account XXXXXXX for more information.",
"errorType": "ModelError",
"requestId": "2f2a7aa4-9eeb-42f5-9a14-6285894581bb",
"stackTrace": [
" File \"/var/task/lambda.py\", line 19, in handler\n response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,\n",
" File \"/var/runtime/botocore/client.py\", line 530, in _api_call\n return self._make_api_call(operation_name, kwargs)\n",
" File \"/var/runtime/botocore/client.py\", line 960, in _make_api_call\n raise error_class(parsed_response, operation_name)\n"
]
}
```
The model configuration is as follow:
- image: 763104351884.dkr.ecr.us-east-2.amazonaws.com/djl-inference:0.24.0-neuronx-sdk2.14.1
- env variables:
- modelId: meta-textgenerationneuron-llama-2-7b-f
- modelVersion: 1.0.0
**To reproduce**
- Deploy the model to an endpoint
- Create a lambda function to query the endpoint with the following code:
```
import boto3
import json
def handler(event, context):
runtime= boto3.client('runtime.sagemaker')
ENDPOINT_NAME = 'testllamaneuron'
dic = {
"inputs": [
[
{"role": "system", "content": "You are chat bot who writes songs"},
{"role": "user", "content": "Write a rap song about Amazon Web Services"}
]
],
"parameters": {"max_new_tokens":256, "top_p":0.9, "temperature":0.6}
}
response = runtime.invoke_endpoint(EndpointName=ENDPOINT_NAME,
ContentType='application/json',
Body=json.dumps(dic),
CustomAttributes="accept_eula=true")
result = json.loads(response['Body'].read().decode())
print(result)
return {
"statusCode": 200,
"body": json.dumps(result)
}
```
**Logs**
Lambda Function logs:
```
[ERROR] ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "BadRequestException",
"message": "Parameter model_name is required."
}
```
Contributor guide
Research direction
Open introduction_to_amazon_algorithms/jumpstart-foundation-models/aws-trainium-inferentia-finetuning-deployment/llama-2-trainium-inferentia-finetuning-deployment.ipynb and review the deployment configuration and invocation payload. Reproduce the Lambda boto3 InvokeEndpoint call against the Llama 2 Neuron endpoint, then trace the reported missing model_name parameter. Done means the documented endpoint invocation completes without the ModelError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- api, cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100