ssd_mobilenet_v2_320x320_coco17_tpu-8 for object detection on GPU using Tensorflow 2 "INFO:tensorflow:depth of additional conv before box predictor: 0" and "WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f51187a02b0>, because it is not built. W0423 10:47:57.954887 139987999496000 save_impl.py:77] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7f511

Prerequisites

• I am using the ssd_mobilenet_v2_320x320_coco17_tpu-8 and TensorFlow 2: 2.3.0
• I am reporting the issue to the correct repository. (research directory) https://github.com/tensorflow/models/blob/master/research/object_detection/configs/tf2/ssd_mobilenet_v2_320x320_coco17_tpu-8.config
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection

2. Describe the bug

Was the error caused by INFO:tensorflow:depth of additional conv before box predictor: 0 and the model after being exported does not use the latest checkpoint (checkpoint3)?

Error messages:
2021-04-23 11:14:58.273174: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-04-23 11:14:59.464301: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-04-23 11:14:59.486935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.487311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2021-04-23 11:14:59.487332: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-04-23 11:14:59.488443: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-04-23 11:14:59.489511: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-04-23 11:14:59.489687: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-04-23 11:14:59.490804: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-04-23 11:14:59.491509: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-04-23 11:14:59.494130: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-04-23 11:14:59.494262: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returni[38/1972]
ode zero
2021-04-23 11:14:59.494699: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.495014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-23 11:14:59.495262: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in perfo
rmance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-04-23 11:14:59.499752: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3600000000 Hz
2021-04-23 11:14:59.500040: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54c78b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-04-23 11:14:59.500051: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-04-23 11:14:59.561614: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.561986: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555b8e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-04-23 11:14:59.562000: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
2021-04-23 11:14:59.562205: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.562522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2021-04-23 11:14:59.562539: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 [18/1972]
2021-04-23 11:14:59.562557: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-04-23 11:14:59.562583: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-04-23 11:14:59.562593: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-04-23 11:14:59.562615: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-04-23 11:14:59.562639: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-04-23 11:14:59.562650: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-04-23 11:14:59.562718: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.563031: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.563370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-04-23 11:14:59.563389: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-04-23 11:14:59.921326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-04-23 11:14:59.921355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0
2021-04-23 11:14:59.921361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N
2021-04-23 11:14:59.921533: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.921864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA n
ode zero
2021-04-23 11:14:59.922150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7014 MB memory) -> physical GPU (device: 0, name:
GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
WARNING:tensorflow:From /home/nguyen_truong/.local/lib/python3.8/site-packages/object_detection/exporter_lib_v2.py:106: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with back_prop=False is deprecated an
d will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
W0423 11:15:00.141795 140359799744320 deprecation.py:568] From /home/nguyen_truong/.local/lib/python3.8/site-packages/object_detection/exporter_lib_v2.py:106: calling map_fn_v2 (from tensorflow.python.ops.map_fn
) with back_prop=False is deprecated and will be removed in a future version.
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
INFO:tensorflow:depth of additional conv before box predictor: 0 [1/1993]
I0423 11:15:03.574930 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0423 11:15:03.575145 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0423 11:15:03.575302 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0423 11:15:03.575448 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0423 11:15:03.575573 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0423 11:15:03.575695 140359799744320 convolutional_keras_box_predictor.py:153] depth of additional conv before box predictor: 0
WARNING:tensorflow:Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7fa7b84d5580>, because it is not built.
W0423 11:15:08.253181 140359799744320 save_impl.py:77] Skipping full serialization of Keras layer <object_detection.meta_architectures.ssd_meta_arch.SSDMetaArch object at 0x7fa7b84d5580>, because it is not built
.
2021-04-23 11:15:15.046676: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
INFO:tensorflow:Assets written to: inference_graph/saved_model/assets
I0423 11:15:26.415863 140359799744320 builder_impl.py:774] Assets written to: inference_graph/saved_model/assets
INFO:tensorflow:Writing pipeline config file to inference_graph/pipeline.config
I0423 11:15:26.901574 140359799744320 config_util.py:253] Writing pipeline config file to inference_graph/pipeline.config

3. Steps to reproduce

Command 1: python model_main_tf2.py --model_dir=ssd_mobilenet_v2_320x320_coco17_tpu-8 --pipeline_config_path=ssd_mobilenet_v2_320x320_coco17_tpu-8/pipeline.config

Command 2: python models/research/object_detection/exporter_main_v2.py --trained_checkpoint_dir ssd_mobilenet_v2_320x320_coco17_tpu-8 --output_directory inference_graph --pipeline_config_path ssd_mobilenet_v2_320x320_coco17_tpu-8/ssd_mobilenet_v2_320x320_coco17_tpu-8.config

4. Expected behavior

Folder inference_graph contained model exported lasted with checkpoint3 and detected images clearly. But the bug exported the model using checkpoint1 has not updated so can not detect images with bounding box.

5. Additional context

My ssd mobilenet v2 configure:
model {
ssd {
num_classes: 1
image_resizer {
fixed_shape_resizer {
height: 300
width: 300
}
}
feature_extractor {
type: "ssd_mobilenet_v2_keras"
depth_multiplier: 1.0
min_depth: 16
conv_hyperparams {
regularizer {
l2_regularizer {
weight: 3.9999998989515007e-05
}
}
initializer {
truncated_normal_initializer {
mean: 0.0
stddev: 0.029999999329447746
}
}
activation: RELU_6
batch_norm {
decay: 0.9700000286102295
center: true
scale: true
epsilon: 0.0010000000474974513
train: true
}
}
override_base_feature_extractor_hyperparams: true
}
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
use_matmul_gather: true
}
}
similarity_calculator {
iou_similarity {
}
}
box_predictor {
convolutional_box_predictor {
conv_hyperparams {
regularizer {
l2_regularizer {
weight: 3.9999998989515007e-05
}
}
initializer {
random_normal_initializer {
mean: 0.0
stddev: 0.009999999776482582
}
}
activation: RELU_6
batch_norm {
decay: 0.9700000286102295
center: true
scale: true
epsilon: 0.0010000000474974513
train: true
}
}
min_depth: 0
max_depth: 0
num_layers_before_predictor: 0
use_dropout: false
dropout_keep_probability: 0.800000011920929
kernel_size: 1
box_code_size: 4
apply_sigmoid_to_scores: false
class_prediction_bias_init: -4.599999904632568
}
}
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.20000000298023224
max_scale: 0.949999988079071
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.33329999446868896
}
}
post_processing {
batch_non_max_suppression {
score_threshold: 9.99999993922529e-09
iou_threshold: 0.6000000238418579
max_detections_per_class: 100
max_total_detections: 100
use_static_shapes: false
}
score_converter: SIGMOID
}
normalize_loss_by_num_matches: true
loss {
localization_loss {
weighted_smooth_l1 {
delta: 1.0
}
}
classification_loss {
weighted_sigmoid_focal {
gamma: 2.0
alpha: 0.75
}
}
classification_weight: 1.0
localization_weight: 1.0
}
encode_background_as_zeros: true
normalize_loc_loss_by_codesize: true
inplace_batchnorm_update: true
freeze_batchnorm: false
}
}
train_config {
batch_size: 12
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
ssd_random_crop {
}
}
sync_replicas: true
optimizer {
momentum_optimizer {
learning_rate {
cosine_decay_learning_rate {
learning_rate_base: 0.03999999910593033
total_steps: 2000
warmup_learning_rate: 0.013333000242710114
warmup_steps: 500
}
}
momentum_optimizer_value: 0.8999999761581421
}
use_moving_average: false
}
fine_tune_checkpoint: "ssd_mobilenet_v2_320x320_coco17_tpu-8/checkpoint0/ckpt-0"
num_steps: 2000
startup_delay_steps: 0.0
replicas_to_aggregate: 8
max_number_of_boxes: 100
unpad_groundtruth_tensors: false
fine_tune_checkpoint_type: "detection"
fine_tune_checkpoint_version: V2
}
train_input_reader {
label_map_path: "data/object-detection.pbxt"
tf_record_input_reader {
input_path: "data/train.record"
}
}
eval_config {
metrics_set: "coco_detection_metrics"
use_moving_averages: false
}
eval_input_reader {
label_map_path: "data/object-detection.pbxt"
shuffle: false
num_epochs: 1
tf_record_input_reader {
input_path: "data/test.record"
}
}

6. System information

• OS Platform and Distribution: Linux Ubuntu 20.04
• TensorFlow installed from (source or binary): pip install
• TensorFlow version (use command below): 2.3.0
• Python version: 3.8.5
• CUDA/cuDNN version: Cuda release 10.1, V10.1.243
• GPU model and memory: GeForce GTX 1070

TensorFlow 2.0
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
--> 2.3.0