Object Detection with Multi GPU training result cannot be inference. Gives error 'Not found: Key BoxPredictor_0/BoxEncodingPredictor/act_quant/max not found in checkpoint'

System information

What is the top-level directory of the model you are using: models/research
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): Source
TensorFlow version (use command below): 1.14.0 gpu
Bazel version (if compiling from source):
CUDA/cuDNN version: Cuda compilation tools, release 10.0, V10.0.130/ CUDNN_MAJOR 7
GPU model and memory: NVIDIA geforce rtx 2080 super X2
Exact command to reproduce:

CUDA_VISIBLE_DEVICES=0,1 python train.py --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v2_quantized_300x300_coco.config --num_clones=2 --ps_tasks=1
python export_inference_graph.py --input_type image_tensor --pipeline_config_path training/ssd_mobilenet_v2_quantized_300x300_coco.config --trained_checkpoint_prefix training/model.ckpt-100 --output_directory inference_graph

Describe the problem

I have followed this link to produce my own object detection. I have successfully completed it with single GPU and I am moving on to use multi GPU.

According to @laksgreen from issue 6611 I have added --num_clones=2 --ps_tasks=1 and increased my batch size to 20. And I have completed the training.

However, when I followed the same process to produce the inference graph, it was failed with the following error

INFO:tensorflow:Restoring parameters from training/model.ckpt-100 I0508 08:01:35.480847 140518557398784 saver.py:1280] Restoring parameters from training/model.ckpt-100 2020-05-08 08:01:36.478474: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key BoxPredictor_0/BoxEncodingPredictor/act_quant/max not found in checkpoint Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call return fn(*args) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.NotFoundError: Key BoxPredictor_0/BoxEncodingPredictor/act_quant/max not found in checkpoint [[{{node save/RestoreV2}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1286, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 950, in run run_metadata_ptr) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run run_metadata) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: Key BoxPredictor_0/BoxEncodingPredictor/act_quant/max not found in checkpoint [[node save/RestoreV2 (defined at /home/docker/models/research/object_detection/exporter.py:341) ]] Original stack trace for 'save/RestoreV2': File "export_inference_graph.py", line 162, in <module> tf.app.run() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "export_inference_graph.py", line 158, in main write_inference_graph=FLAGS.write_inference_graph) File "/home/docker/models/research/object_detection/exporter.py", line 510, in export_inference_graph write_inference_graph=write_inference_graph) File "/home/docker/models/research/object_detection/exporter.py", line 439, in _export_inference_graph trained_checkpoint_prefix=checkpoint_to_use) File "/home/docker/models/research/object_detection/exporter.py", line 341, in write_graph_and_checkpoint tf.import_graph_def(inference_graph_def, name='') File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def _ProcessNewOps(graph) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3751, in <listcomp> for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation ret = Operation(c_op, self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__ self._traceback = tf_stack.extract_stack() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1296, in restore names_to_keys = object_graph_key_mapping(save_path) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1614, in object_graph_key_mapping object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 678, in get_tensor return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str)) tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint During handling of the above exception, another exception occurred: Traceback (most recent call last): File "export_inference_graph.py", line 162, in <module> tf.app.run() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "export_inference_graph.py", line 158, in main write_inference_graph=FLAGS.write_inference_graph) File "/home/docker/models/research/object_detection/exporter.py", line 510, in export_inference_graph write_inference_graph=write_inference_graph) File "/home/docker/models/research/object_detection/exporter.py", line 439, in _export_inference_graph trained_checkpoint_prefix=checkpoint_to_use) File "/home/docker/models/research/object_detection/exporter.py", line 345, in write_graph_and_checkpoint saver.restore(sess, trained_checkpoint_prefix) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1302, in restore err, "a Variable name or other graph key that is missing") tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: Key BoxPredictor_0/BoxEncodingPredictor/act_quant/max not found in checkpoint [[node save/RestoreV2 (defined at /home/docker/models/research/object_detection/exporter.py:341) ]] Original stack trace for 'save/RestoreV2': File "export_inference_graph.py", line 162, in <module> tf.app.run() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 299, in run _run_main(main, args) File "/usr/local/lib/python3.5/dist-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "export_inference_graph.py", line 158, in main write_inference_graph=FLAGS.write_inference_graph) File "/home/docker/models/research/object_detection/exporter.py", line 510, in export_inference_graph write_inference_graph=write_inference_graph) File "/home/docker/models/research/object_detection/exporter.py", line 439, in _export_inference_graph trained_checkpoint_prefix=checkpoint_to_use) File "/home/docker/models/research/object_detection/exporter.py", line 341, in write_graph_and_checkpoint tf.import_graph_def(inference_graph_def, name='') File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def _ProcessNewOps(graph) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3751, in <listcomp> for c_op in c_api_util.new_tf_operations(self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation ret = Operation(c_op, self) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__ self._traceback = tf_stack.extract_stack()

It looks like the key cannot found in checkpoint.
I have searched for some information about
Not found: Key BoxPredictor_0/BoxEncodingPredictor/act_quant/max not found in checkpoint
in this issue 1783 @roadcode has mentioned the check point node name problem. When training with multi GPU, there I found that there are some node name started with clone_0 and clone_1, but I do not know how to rewrite the node name. Is it correct to rename the nodes?