Context R-CNN Model Evaluation Raises "ValueError: Exception encountered when calling layer 'AttentionBlock' (type AttentionBlock)"

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/research/object_detection/model_main_tf2.py

2. Describe the bug

INFO:tensorflow:Encountered in user code:

File "/usr/local/lib/python3.10/dist-packages/object_detection/model_lib_v2.py", line 910, in compute_eval_dict  *
    losses_dict, prediction_dict = _compute_losses_and_predictions_dicts(
File "/usr/local/lib/python3.10/dist-packages/object_detection/model_lib_v2.py", line 124, in _compute_losses_and_predictions_dicts  *
    prediction_dict = model.predict(
File "/usr/local/lib/python3.10/dist-packages/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 823, in predict  *
    prediction_dict.update(
File "/usr/local/lib/python3.10/dist-packages/object_detection/meta_architectures/context_rcnn_meta_arch.py", line 430, in _predict_second_stage  *
    prediction_dict = self._box_prediction(rpn_features_to_crop,
File "/usr/local/lib/python3.10/dist-packages/object_detection/meta_architectures/context_rcnn_meta_arch.py", line 500, in _box_prediction  *
    box_classifier_features = self._extract_box_classifier_features(
File "/usr/local/lib/python3.10/dist-packages/object_detection/meta_architectures/context_rcnn_meta_arch.py", line 614, in _extract_box_classifier_features  *
    attention_features = self._context_feature_extract_fn(
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler  **
    raise e.with_traceback(filtered_tb) from None
File "/tmp/__autograph_generated_file_2rr8_ae.py", line 29, in tf__call
    box_features = ag__.converted_call(ag__.ld(tf).reshape, (ag__.ld(box_features), [ag__.ld(batch_size), ag__.ld(self)._max_num_proposals, ag__.ld(height), ag__.ld(width), ag__.ld(channels)]), None, fscope)

ValueError: Exception encountered when calling layer 'AttentionBlock' (type AttentionBlock).

in user code:

    File "/usr/local/lib/python3.10/dist-packages/object_detection/meta_architectures/context_rcnn_lib_tf2.py", line 118, in call  *
        box_features = tf.reshape(

    ValueError: Cannot reshape a tensor with 39321600 elements to shape [18,64,4,4,2048] (37748736 elements) for '{{node AttentionBlock/Reshape}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](model_1/conv5_block3_out/Relu, AttentionBlock/Reshape/shape)' with input shapes: [1200,4,4,2048], [5] and with input tensors computed as partial shapes: input[1] = [18,64,4,4,2048].


Call arguments received by layer 'AttentionBlock' (type AttentionBlock):
  • box_features=tf.Tensor(shape=(1200, 4, 4, 2048), dtype=float32)
  • context_features=tf.Tensor(shape=(4, 2000, 2057), dtype=float32)
  • valid_context_size=tf.Tensor(shape=(4,), dtype=int32)
  • num_proposals=tf.Tensor(shape=(4,), dtype=int32)
INFO:tensorflow:A replica probably exhausted all examples. Skipping pending examples on other replicas.
I0726 18:38:16.505056 133500551127040 model_lib_v2.py:943] A replica probably exhausted all examples. Skipping pending 
examples on other replicas.
Traceback (most recent call last):
   File "/content/models/research/object_detection/model_main_tf2.py", line 114, in <module>
     tf.compat.v1.app.run()
   File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/platform/app.py", line 36, in run
     _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
   File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 308, in run
     _run_main(main, args)
   File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 254, in _run_main
     sys.exit(main(argv))
   File "/content/models/research/object_detection/model_main_tf2.py", line 81, in main
     model_lib_v2.eval_continuously(
   File "/usr/local/lib/python3.10/dist-packages/object_detection/model_lib_v2.py", line 1158, in eval_continuously
     eager_eval_loop(
   File "/usr/local/lib/python3.10/dist-packages/object_detection/model_lib_v2.py", line 1009, in eager_eval_loop
     for evaluator in evaluators:
 TypeError: 'NoneType' object is not iterable

3. Steps to reproduce

I am trying to run eval for context r-cnn model using the config file from: 'https://github.com/tensorflow/models/research/object_detection/test_data/context_rcnn_camera_trap.config'. I am using the same config file for training the context r-cnn model which is running completely fine. But the evaluation process raises the above error that I have provided. It seems to me that the model is encountering an issue while calling the "AttentionBlock" layer, specifically when trying to reshape tensors. It appears that the model is trying to reshape a tensor with 39,321,600 elements to a shape of [18, 64, 4, 4, 2048], but the new shape would have 37,748,736 elements, which is not compatible with the original tensor's size. The 39,321,600 elements are coming from the box_features tensor which is of the shape (1200, 4, 4, 2048) such that: 1200 * 4 * 4 * 2048 = 39,321,600 elements.

4. Expected behavior

I am confused as to why this issue is arising only during the model validation stage and not during training stage since the scripts for both of these stages should be quite similar.

5. Additional context

For the tfrecords, I have exported standard TfExamples and not TF_SEQUENCE_EXAMPLE. I have also verified the fact that there is no issue with the tfrecord files by setting the input_path for validation inside the config file with the training tfrecord files path. Since the training is running fine with the same train tfrecord files, then there must be no issue with the train tfrecord files and so the issue should not have been arisen for the same tfrecord files during the evaluation stage.