tensorflow/serving:1.15.0 saved_model_warmup.cc:105] No warmup data file found at s3
Nobody has claimed this yet.
- Dominant language
- Jsonnet
- Stars
- 1.5k
- Forks
- 752
- PR merge metrics
- No merged PRs in 30d
Description
I have followed process in mnist_vanilla_k8s.ipynb , everything is ok. But when deploy serving model,
i got errors
`2022-10-13 05:01:41.524629: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:105] No warmup data file found at s3://lo-harbor-mnist/mnist/export/1665634166/assets.extra/tf_serving_warmup_requests
2022-10-13 05:01:41.524693: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2022-10-13 05:01:41.524872: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
2022-10-13 05:01:41.525581: E external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:60] HTTP response code: 404
Exception name:
Error message: No response body.
8 response headers:
accept-ranges : bytes
content-length : 0
content-security-policy : block-all-mixed-content
date : Thu, 13 Oct 2022 05:01:41 GMT
server : MinIO/RELEASE.2019-08-14T20-37-41Z
vary : Origin
x-amz-request-id : 171D884D020B07AE
x-xss-protection : 1; mode=block`
I check s3 bucket, there is not exist warmup file mention above s3://lo-harbor-mnist/mnist/export/1665634166/assets.extra/tf_serving_warmup_requests
Environment:
Kubenetes version:v1.21.14
kubeflow version:1.5.1
jupyter notebook version:public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:v1.5.0
mnist image version: tensorflow/tensorflow:1.15.2-py3
modle serving version:tensorflow/serving:1.15.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the deployment described in mnist_vanilla_k8s.ipynb and inspect the exported model's assets.extra path together with the TensorFlow Serving warmup log at saved_model_warmup.cc:105. Compare the expected export contents with the S3/MinIO object path and document a reproducible result, including whether the missing warmup file is expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, tensorflow
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100