aws / aws/amazon-sagemaker-examples

[Bug Report]

Open
#4,336 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/question_answering_retrieval_augmented_generation/question_answering_langchain_jumpstart.ipynb

**Describe the bug**
Invoking the endpoints after deployment is not working. I keep getting the following model error:
```
---------------------------------------------------------------------------
ModelError Traceback (most recent call last)
Cell In[55], line 16
14 for model_id in list_of_LLMs:
15 endpoint_name = _MODEL_CONFIG_[model_id]["endpoint_name"]
---> 16 query_response = query_endpoint_with_json_payload(
17 json.dumps(payload).encode("utf-8"), endpoint_name=endpoint_name, offload_folder=offload_folder
18 )
19 generated_texts = _MODEL_CONFIG_[model_id]["parse_function"](query_response)
20 print(f"For model: {model_id}, the generated output is: {generated_texts[0]}\n")

Cell In[51], line 4, in query_endpoint_with_json_payload(encoded_json, endpoint_name, content_type, offload_folder)
2 client = boto3.client("runtime.sagemaker")
3 if offload_folder is not None:
----> 4 response = client.invoke_endpoint(
5 EndpointName=endpoint_name, ContentType=content_type, Body=encoded_json, CustomAttributes=f'{{"offload_folder":"{offload_folder}"}}'
6 )
7 else:
8 response = client.invoke_endpoint(
9 EndpointName=endpoint_name,
10 ContentType=content_type,
11 Body=encoded_json
12 )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:530, in ClientCreator._create_api_method.._api_call(self, *args, **kwargs)
526 raise TypeError(
527 f"{py_operation_name}() only accepts keyword arguments."
528 )
529 # The "self" in this scope is referring to the BaseClient.
--> 530 return self._make_api_call(operation_name, kwargs)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:964, in BaseClient._make_api_call(self, operation_name, api_params)
962 error_code = parsed_response.get("Error", {}).get("Code")
963 error_class = self.exceptions.from_code(error_code)
--> 964 raise error_class(parsed_response, operation_name)
965 else:
966 return parsed_response

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "InternalServerException",
"message": "At least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`."
}
". See https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/jumpstart-example-raglc-huggingface-tex-2023-07-23-04-36-50-245 in account 367332732143 for more information.

```

**To reproduce**
The notebook instance is set to `ml.g5.xlarge`

**Logs**
If applicable, add logs to help explain your problem.
Here is a screenshot of the section of notebook showing the code and the error:
![image](https://github.com/aws/amazon-sagemaker-examples/assets/123378149/713f6779-a7f3-4c1a-8819-20c52d2c997c)

Contributor guide

Open the contributing guide

Research direction

Start with the linked question_answering_langchain_jumpstart.ipynb, focusing on query_endpoint_with_json_payload and the endpoint invocation cell. Reproduce the ModelError on an ml.g5.xlarge notebook instance and compare the request with the logged offload_folder requirement. Done means the deployed endpoints can be invoked successfully from the notebook.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.