aws / aws/sagemaker-tensorflow-extensions
Sagemaker endpoint fails to respond then PipeModeDataset is used
- Dominant language
- C++
- Stars
- 54
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
Sagemaker endpoint fails to respond to the requests with the following error:
2019-11-06 13:21:06.753786: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name:
model version: 1} failed: Not found: Op type not registered 'PipeModeDataset' in binary running
on model.aws.local. Make sure the Op and Kernel are registered in the binary running in this
process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing
(e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily
registered when the module is first accessed.
14:21:20 2019/11/06 13:21:20 [error] 23#23: *266 connect() failed (111: Connection refused)
while connecting to upstream, client: 10.32.0.2, server: , request: "POST /invocations HTTP/1.1",
subrequest: "/v1/models/model:predict", upstream:
"http://127.0.0.1:22001/v1/models/model:predict", host: "model.aws.local:8080"
Contributor guide
Research direction
Start with the reported SageMaker endpoint logs and the PipeModeDataset registration error. Trace how the model is loaded by the serving process and verify whether the serving binary supports the operation used by the saved graph. Done means the endpoint responds successfully to inference requests without the reported loading or connection errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100