tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid argument: Reduction axis 1 is empty in shape [1,0]

Prerequisites

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not been filed already.

System information:

Top-level directory of the model: object_detection
OS Platform and Distribution: Windows 10
TensorFlow installed from (source or binary): source
TensorFlow version: 1.15
CUDA/cuDNN version: 9.2
GPU model and memory: RTX6000
Python version: 3.6.10

Error description and error log

I am currently training a Faster RCNN with resnet50 on a custom dataset using Tensorlow 1.15 (pulled from branch r1.13.0) using cuda 9.2 and RTX6000. Here is some more information:

I get to train the model several times and sometimes it prints out this error:

Traceback (most recent call last):
  File "script.py", line 321, in <module>
    tf.app.run()
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "script.py", line 317, in main
    study.optimize(objective, n_trials=FLAGS.n_trials_)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/optuna/study.py", line 292, in optimize
    func, n_trials, timeout, catch, callbacks, gc_after_trial, None
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/optuna/study.py", line 654, in _optimize_sequential
    self._run_trial_and_callbacks(func, catch, callbacks, gc_after_trial)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/optuna/study.py", line 685, in _run_trial_and_callbacks
    trial = self._run_trial(func, catch, gc_after_trial)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/optuna/study.py", line 709, in _run_trial
    result = func(trial)
  File "script.py", line 309, in objective
    FLAGS.run_once)
  File "../model_main_fn.py", line 85, in model_main_fn
    eval_result, _ = tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
  return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Reduction axis 1 is empty in shape [1,0]
         [[node Loss/RPNLoss/Match/cond/ArgMax_1 (defined at /home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]                               [[BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Greater/_3911]]
  (1) Invalid argument: Reduction axis 1 is empty in shape [1,0]
         [[node Loss/RPNLoss/Match/cond/ArgMax_1 (defined at /home/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

I tried to use:
fixed_shape_resizer {height:250 widith:5000}
instead of the default configuration:
keep_aspect_ratio_resizer {min_dimension: 600 max_dimension: 1024}
but it didn't really help.

Does anybody know what is causing this error?