kubeflow / kubeflow/examples

tensorflow/serving:1.15.0 saved_model_warmup.cc:105] No warmup data file found at s3

Open
#1,002 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jsonnet
Stars
1.5k
Forks
752
PR merge metrics
No merged PRs in 30d

Description

I have followed process in mnist_vanilla_k8s.ipynb , everything is ok. But when deploy serving model,
i got errors

`2022-10-13 05:01:41.524629: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:105] No warmup data file found at s3://lo-harbor-mnist/mnist/export/1665634166/assets.extra/tf_serving_warmup_requests
2022-10-13 05:01:41.524693: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2022-10-13 05:01:41.524872: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
2022-10-13 05:01:41.525581: E external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:60] HTTP response code: 404
Exception name: 
Error message: No response body.
8 response headers:
accept-ranges : bytes
content-length : 0
content-security-policy : block-all-mixed-content
date : Thu, 13 Oct 2022 05:01:41 GMT
server : MinIO/RELEASE.2019-08-14T20-37-41Z
vary : Origin
x-amz-request-id : 171D884D020B07AE
x-xss-protection : 1; mode=block`

I check s3 bucket, there is not exist warmup file mention above s3://lo-harbor-mnist/mnist/export/1665634166/assets.extra/tf_serving_warmup_requests

Environment:
Kubenetes version:v1.21.14
kubeflow version:1.5.1
jupyter notebook version:public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:v1.5.0
mnist image version: tensorflow/tensorflow:1.15.2-py3
modle serving version:tensorflow/serving:1.15.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the deployment described in mnist_vanilla_k8s.ipynb and inspect the exported model's assets.extra path together with the TensorFlow Serving warmup log at saved_model_warmup.cc:105. Compare the expected export contents with the S3/MinIO object path and document a reproducible result, including whether the missing warmup file is expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, tensorflow
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.