tensorflow / tensorflow/models

Unable to get batched input argument from exported Object Detection Saved Model for TensorFlow Batch Serving

Open
#10,552 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

models:research type:bug
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_lib_v2.py

I am using TensorFlow-2 Object Detection API to build my custom object detection model using FasterRCNN Resnet-152 640x640 Pretrained Model from the tf2 Model Zoo. I have successfully trained my model and the checkpoints created are in below format -
image
I then used [exporter_main_v2.py] (https://github.com/tensorflow/models/blob/master/research/object_detection/exporter_main_v2.py) to export the model. I used --input_type as 'image_tensor' the first time and 'float_image_tensor' the second time. After running the script saved_model.pb was generated and, the output folder looked like this -
image
image

To get the meta data information, I ran the saved_model_cli command and this is how it looks -
image

I used the same saved_model.pb for inferencing using TF Serving command -
tensorflow_model_server --port=8500 --model_name=detection_model --model_base_path=path_to_saved_model(/tfserve_savedmodel/)
I was able to get the results for single image but cannot perform batch inferencing.

Points to consider-

  1. I am using GRPC client to establish the connection to the server and I updated the code for batch inferencing. I was passing a batch of numpy array of size (1,640,640,3) for single image and tried creating a np array (10,640,640,3) for a batch of 10 images and passing it to the server.
  2. I thought the input shape from the SavedModel SignatureDef had shape: (1, -1, -1, 3) and this is the reason I wasn't able to do for multiple images. Thus I updated the code in file exporter_lib_v2.py and set the shape to shape=[None, None, None, 3] and exported the model. But even then I got the input shape as (1, -1, -1, 3).
  3. To dig deeper I checked the metadata information of the Pretrained Model(Faster R-CNN ResNet152 V1 640x640) from model zoo(https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md) I used for training my model. It also has the same SignatureDef input shape: (1, -1, -1, 3). I suspect if this is the reason my final saved model has batchsize of 1 even after updating the exporter_lib_v2.py file. I am facing issues to perform batch inferencing.
  4. Also I checked the Tensorflow Serving Github Repo (https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/serving_config.md#batching-configuration) and tried passing a separate batch.config while running the server but still could not perform batch inferencing. I used the below command - tensorflow_model_server --port=8500 --model_config_file=/data1/root/lab/prime_team_projects/scripts/models.config.a --enable_batching true --batching_parameters_file=/data1/root/lab/prime_team_projects/scripts/batch.config

Help me with this issue please.

System information

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with exporter_main_v2.py and exporter_lib_v2.py, then inspect the exported signature with saved_model_cli. Reproduce the request through the gRPC client and TensorFlow Serving with the cited batching configuration. Done means determining whether the exported model or serving setup prevents batched inputs and documenting the confirmed behavior or required change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
backend-api-design, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.