aws / aws/amazon-sagemaker-examples

[Bug Report] ModelError when calling the InvokeEndpoint operation

Open
#2,815 3 comments 8 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-fulltraining-elastic-inference.ipynb

**Describe the bug**
The endpoint is working without any problems, but sometimes, the endpoint suddenly stops working and any request to the endpoint resolves to the following error:

`An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from model with message "Amazon SageMaker could not get a response from the [ENDPOINT NAME] endpoint. This can occur when CPU or memory utilization is high. To check your utilization, see Amazon CloudWatch. To fix this problem, use an instance type with more CPU capacity or memory.". See https://eu-central-1.console.aws.amazon.com/cloudwatch/home?region=eu-central-1#logEventViewer:group=/aws/sagemaker/Endpoints/[ENDPOINT NAME] in account XXXXXXXXXXXX for more information.`

It seems as if the instance type runs out of memory or CPU, but I have performed stress tests on the endpoint it handles many requests without any problem. I have monitored the memory and CPU during the stress tests and neither of them go over 10%. The endpoint logs show that the workers are "abnormally terminated" and new workers are instantiated, but these are terminated as well. (full logs attached)

I want to be able to control this error and understand why is it happening and how to prevent it.

**To reproduce**
I can't find a way to reproduce this error as this happens all of a sudden in the endpoint instance.

**Logs**
[Enpoint logs](https://diatomea-my.sharepoint.com/:x:/g/personal/e_fernandez_diatomea_tech/EdEMkZlvu1dOuF42fWtcnkwBKsxqFneF1CCDv4EGInwa4w?e=vxLehX)

Contributor guide

Open the contributing guide

Research direction

Start with the linked Image-classification-fulltraining-elastic-inference.ipynb, then review the SageMaker endpoint logs and the referenced CloudWatch endpoint logs alongside the reported stress-test observations. The issue has no reproducible steps; the investigation is complete only when the cause of the abnormally terminated workers and a way to control or prevent the ModelError are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
cloud, machine-learning, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.