Error When Evaluating Object Detection Model: The two structures don't have the same nested structure.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py

2. Describe the bug

I tried training a custom object detector using the TFOD API's pre-trained faster rcnn resnet101 model using TF2.5.0. Then, I tried evaluating the model as explained in the docs. However, when I tried running the command to run model_main_tf2.py with the --checkpoint_dir flag, I get this error:

Traceback (most recent call last):
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 526, in assert_same_structure
    expand_composites)
ValueError: The two structures don't have the same nested structure.

First structure: type=DType str=<dtype: 'uint8'>

Second structure: type=list str=[]

More specifically: Substructure "type=list str=[]" is a sequence, while substructure "type=DType str=<dtype: 'uint8'>" is not

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_main_tf2.py", line 119, in <module>
    tf.compat.v1.app.run()
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 94, in main
    wait_interval=300, timeout=FLAGS.eval_timeout)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/object_detection/model_lib_v2.py", line 1157, in eval_continuously
    global_step=global_step,
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/object_detection/model_lib_v2.py", line 968, in eager_eval_loop
    keypoint_edges=keypoint_edges or None)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/object_detection/utils/visualization_utils.py", line 758, in draw_side_by_side_evaluation_image
    use_normalized_coordinates=use_normalized_coordinates)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/object_detection/utils/visualization_utils.py", line 649, in draw_bounding_boxes_on_image_tensors
    images = tf.map_fn(draw_boxes, elems, dtype=tf.uint8, back_prop=False)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 535, in new_func
    return func(*args, **kwargs)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/map_fn.py", line 507, in map_fn
    maximum_iterations=n)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2777, in while_loop
    loop_vars = body(*loop_vars)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2768, in <lambda>
    body = lambda i, lv: (i + 1, orig_body(*lv))
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/ops/map_fn.py", line 492, in compute
    nest.assert_same_structure(fn_output_signature or elems, result_value)
  File "/home/nardiena/.virtualenvs/tfod-api-gpu-2.5.0/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 533, in assert_same_structure
    % (str(e), str1, str2))
ValueError: The two structures don't have the same nested structure.

First structure: type=DType str=<dtype: 'uint8'>

Second structure: type=list str=[]

More specifically: Substructure "type=list str=[]" is a sequence, while substructure "type=DType str=<dtype: 'uint8'>" is not
Entire first structure:
.
Entire second structure:
[]

3. Steps to reproduce

• Train the model by running: python model_main_tf2.py --model_dir=/path/to/model/dir --pipeline_config_path=/path/to/config/file

• After training, run the above command again but with --checkpoint_dir flag: python model_main_tf2.py --model_dir=/path/to/model/dir --pipeline_config_path=/path/to/config/file --checkpoint_dir=/path/to/model/dir

4. Expected behavior

What I expected to happen was the evaluation process would start running and the file would use the checkpoints to evaluate the performance of the model. The output was supposed to look like the one in the docs.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v2.5.0-rc3-213-ga4dfb8d1a71 2.5.0
• Python version: 3.7.0
• CUDA/cuDNN version: 11.2/8.1
• GPU model and memory: NVIDIA RTX 2060 Mobile, 3GB