aws / aws/amazon-sagemaker-examples
[Bug Report]
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/question_answering_retrieval_augmented_generation/question_answering_langchain_jumpstart.ipynb
**Describe the bug**
Invoking the endpoints after deployment is not working. I keep getting the following model error:
```
---------------------------------------------------------------------------
ModelError Traceback (most recent call last)
Cell In[55], line 16
14 for model_id in list_of_LLMs:
15 endpoint_name = _MODEL_CONFIG_[model_id]["endpoint_name"]
---> 16 query_response = query_endpoint_with_json_payload(
17 json.dumps(payload).encode("utf-8"), endpoint_name=endpoint_name, offload_folder=offload_folder
18 )
19 generated_texts = _MODEL_CONFIG_[model_id]["parse_function"](query_response)
20 print(f"For model: {model_id}, the generated output is: {generated_texts[0]}\n")
Cell In[51], line 4, in query_endpoint_with_json_payload(encoded_json, endpoint_name, content_type, offload_folder)
2 client = boto3.client("runtime.sagemaker")
3 if offload_folder is not None:
----> 4 response = client.invoke_endpoint(
5 EndpointName=endpoint_name, ContentType=content_type, Body=encoded_json, CustomAttributes=f'{{"offload_folder":"{offload_folder}"}}'
6 )
7 else:
8 response = client.invoke_endpoint(
9 EndpointName=endpoint_name,
10 ContentType=content_type,
11 Body=encoded_json
12 )
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:530, in ClientCreator._create_api_method.._api_call(self, *args, **kwargs)
526 raise TypeError(
527 f"{py_operation_name}() only accepts keyword arguments."
528 )
529 # The "self" in this scope is referring to the BaseClient.
--> 530 return self._make_api_call(operation_name, kwargs)
File ~/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py:964, in BaseClient._make_api_call(self, operation_name, api_params)
962 error_code = parsed_response.get("Error", {}).get("Code")
963 error_class = self.exceptions.from_code(error_code)
--> 964 raise error_class(parsed_response, operation_name)
965 else:
966 return parsed_response
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
"code": 400,
"type": "InternalServerException",
"message": "At least one of the model submodule will be offloaded to disk, please pass along an `offload_folder`."
}
". See https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/aws/sagemaker/Endpoints/jumpstart-example-raglc-huggingface-tex-2023-07-23-04-36-50-245 in account 367332732143 for more information.
```
**To reproduce**
The notebook instance is set to `ml.g5.xlarge`
**Logs**
If applicable, add logs to help explain your problem.
Here is a screenshot of the section of notebook showing the code and the error:

Contributor guide
Research direction
Start with the linked question_answering_langchain_jumpstart.ipynb, focusing on query_endpoint_with_json_payload and the endpoint invocation cell. Reproduce the ModelError on an ml.g5.xlarge notebook instance and compare the request with the logged offload_folder requirement. Done means the deployed endpoints can be invoked successfully from the notebook.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100