aws / aws/amazon-sagemaker-examples

Received server error (500) from primary and could not load the entire response body from endpoint

Open
#4,324 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

[ERROR] ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from primary and could not load the entire response body. See https://eu-west-3.console.aws.amazon.com/cloudwatch/home?region=eu-west-3#logEventViewer:group=/aws/sagemaker/Endpoints/pytorch-inference-2023-07-10-09-51-02-299 in account 086892845792 for more information. Traceback (most recent call last): File "/var/task/lambda_function.py", line 442, in lambda_handler pred_prob = invoke_endpoint_with_idx(endpointname = ENDPOINT_NAME, target_id = transaction_id, subgraph_dict = subgraph_dict, n_feats = transaction_embed_value_dict) File "/var/task/lambda_function.py", line 314, in invoke_endpoint_with_idx response = runtime.invoke_endpoint(EndpointName=endpointname, File "/opt/python/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "/opt/python/lib/python3.8/site-packages/botocore/client.py", line 911, in _make_api_call raise error_class(parsed_response, operation_name) [enter image description here](https://i.stack.imgur.com/S5H8f.png) please i got this error while running the following code https://github.com/awslabs/realtime-fraud-detection-with-gnn-on-dgl/tree/main/src/sagemaker. once shouted the endpoint and called it from lambda I received that, I redid several times it's always the same error, can you help me please

I run the preprocessing code then train the model on sagemaker , after that i deploy it ( the code is from aws : https://github.com/awslabs/realtime-fraud-detection-with-gnn-on-dgl/tree/main/src/sagemaker) i invoke the endpoint with lambda and i receive this server error, in the first it's was ok then it's became like this , I tried everything but still the same problem,
![image](https://github.com/aws/amazon-sagemaker-examples/assets/119299684/09263115-b537-4c13-a2f7-19495896422d)

Contributor guide

Open the contributing guide

Research direction

Start with the SageMaker example under src/sagemaker and inspect lambda_function.py at the invocation path around lines 314 and 442. Check the linked CloudWatch endpoint logs alongside the Lambda request and response, then compare them with the example's deployment and invocation flow. Done means the deployed endpoint can be invoked from Lambda without the 500 response and the full response body is received.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.