tensorflow / tensorflow/serving

Cannot load saved model from S3 bucket after tensorflow-serving 2.7

Open
#1,963 23 comments 5 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

stat:awaiting tensorflower type:bug
Dominant language
C++
Stars
6.4k
Forks
2.2k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

If this is a bug report, please fill out the following form in full:

System information
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  • TensorFlow Serving installed from (source or binary): Docker image (tensorflow/serving:2.7.0)
  • TensorFlow Serving version: 2.7.0
Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in
TensorFlow Serving.

SavedModel cannot be loaded from S3 bucket after tensorflow-serving 2.7. Server raised error like below:

2022-01-18 02:30:30.032712: E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:365] FileSystemStoragePathSource encountered a filesystem access error: Could not find base path s3://some-s3-path..... for servable model with error UNIMPLEMENTED: File system scheme 's3' not implemented (file: 's3://some-s3-path.....')

Docker image tensorflow/serving:2.6.2 runs without any error in same configuration.

Exact Steps to Reproduce

Please include all steps necessary for someone to reproduce this issue on their
own machine. If not, skip this section.

docker run --rm -it \
  --env AWS_REGION=some-aws-region \
  --env MODEL_BASE_PATH=some-s3-path \
  tensorflow/serving:2.7.0
Source code / logs

Include any logs or source code that would be helpful to diagnose the problem.
If including tracebacks, please include the full traceback. Large logs and files
should be attached. Try to provide a reproducible test case that is the bare
minimum necessary to generate the problem.

2022-01-18 02:30:30.032712: E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:365] FileSystemStoragePathSource encountered a filesystem access error: Could not find base path s3://some-s3-path..... for servable model with error UNIMPLEMENTED: File system scheme 's3' not implemented (file: 's3://some-s3-path.....')

I think this is because of Modular File System Migration in TF 2.7.0's release note. Is there any way to link tensorflow-io in the build step?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.