aws / aws/sagemaker-tensorflow-extensions

Sagemaker endpoint fails to respond then PipeModeDataset is used

Open
#55 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
54
Forks
39
PR merge metrics
No merged PRs in 30d

Description

Sagemaker endpoint fails to respond to the requests with the following error:

2019-11-06 13:21:06.753786: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name:
model version: 1} failed: Not found: Op type not registered 'PipeModeDataset' in binary running
on model.aws.local. Make sure the Op and Kernel are registered in the binary running in this
process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing
(e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily
registered when the module is first accessed.

14:21:20 2019/11/06 13:21:20 [error] 23#23: *266 connect() failed (111: Connection refused)
while connecting to upstream, client: 10.32.0.2, server: , request: "POST /invocations HTTP/1.1",
subrequest: "/v1/models/model:predict", upstream:
"http://127.0.0.1:22001/v1/models/model:predict", host: "model.aws.local:8080"

Contributor guide

Open the contributing guide

Research direction

Start with the reported SageMaker endpoint logs and the PipeModeDataset registration error. Trace how the model is loaded by the serving process and verify whether the serving binary supports the operation used by the saved graph. Done means the endpoint responds successfully to inference requests without the reported loading or connection errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.