alibaba / alibaba/EasyRec

Failed to start server. Error: Unknown: 1 servable(s) did not become available

Open
#373 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.4k
Forks
383
PR merge metrics
No merged PRs in 30d

Description

`docker run -p 8501:8501 -p 8500:8500 --mount type=bind,source=/model/test,target=/models/test -e MODEL_NAME=test -t tensorflow/serving:2.1.0`

2023-05-24 10:00:07.041263: I tensorflow_serving/util/retrier.cc:33] Retrying of Loading servable: {name: test version: 1} retry: 5
2023-05-24 10:00:07.049557: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/test/1
2023-05-24 10:00:07.071548: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2023-05-24 10:00:07.071613: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:264] Reading SavedModel debug info (if present) from: /models/test/1
2023-05-24 10:00:07.085314: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:333] SavedModel load for tags { serve }; Status: fail: Not found: Op type not registered 'StatelessRandomGetKeyCounter' in binary running on 3e5ec75a2724. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.. Took 35764 microseconds.
2023-05-24 10:00:07.085387: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: test version: 1} failed: Not found: Op type not registered 'StatelessRandomGetKeyCounter' in binary running on 3e5ec75a2724. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
2023-05-24 10:00:07.085449: I tensorflow_serving/util/retrier.cc:46] Retrying of Loading servable: {name: test version: 1} exhausted max_num_retries: 5
2023-05-24 10:00:07.085536: I tensorflow_serving/core/loader_harness.cc:155] Encountered an error for servable version {name: test version: 1}: Not found: Op type not registered 'StatelessRandomGetKeyCounter' in binary running on 3e5ec75a2724. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
2023-05-24 10:00:07.085573: E tensorflow_serving/core/aspired_versions_manager.cc:359] Servable {name: test version: 1} cannot be loaded: Not found: Op type not registered 'StatelessRandomGetKeyCounter' in binary running on 3e5ec75a2724. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
Failed to start server. Error: Unknown: 1 servable(s) did not become available: {{{name: test version: 1} due to error: Not found: Op type not registered 'StatelessRandomGetKeyCounter' in binary running on 3e5ec75a2724. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.}, }
2023-05-24 10:00:07.768893: I tensorflow_serving/core/basic_manager.cc:280] Unload all remaining servables in the manager.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the `docker run` command from the issue with the referenced SavedModel and inspect the TensorFlow Serving logs. Compare the model's required op, `StatelessRandomGetKeyCounter`, with the `tensorflow/serving:2.1.0` runtime. Done means the compatibility cause and a verified resolution or supported version combination are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, tensorflow
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.