Problem with image decoding in Tensorflow 2

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [Y ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [Y ] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [Y ] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection

2. Describe the bug

I am trying to train the model using Tensorflow 2. The pipeline seems to work (training starts and the training process seems to be running), but I noticed a disturbing symptom - in Tensorboard the preview of images is incorrect - they look like they are badly decoded (color values truncated to 0 and 1 - example below).

From what I remember - when I was using Object Detection API with TF1, the preview displayed "normal" images.
I am not sure if this is a bug related to Tesnsorboard visualization only or if the training pipeline does not work as it should and the images are loaded incorrectly. Or maybe I am making a configuration mistake?

3. Steps to reproduce

• Generate tfrecords for the PASCAL VOC file according to the instructions in the https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/preparing_inputs.md (to make sure that the data is not the problem).
• Create a config based on any of the available configs, e.g. faster_rcnn_resnet50_v1_fpn_640x640_coco17_tpu-8.config (change num_classes, use_bfloat16, fine_tune_checkpoint and paths to generated tfrecords and appropriate label_map in input_readers)
• Start local training

# From the tensorflow/models/research/ directory
PIPELINE_CONFIG_PATH={path to pipeline config file}
MODEL_DIR={path to model directory}
python object_detection/model_main_tf2.py \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --model_dir=${MODEL_DIR} \
    --alsologtostderr

• Launch Tensorboard

tensorboard --logdir=${MODEL_DIR}

4. Expected behavior

I expect to see properly decoded images.

5. Additional context

Suspect part of the training log (but I'm not sure if it has anything to do with the issue):

WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0816 20:24:48.906076 140309352765248 dataset_builder.py:83] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
W0816 20:24:48.957060 140309352765248 image_ops_impl.py:2018] The operation `tf.image.convert_image_dtype` will be skippedsince the input and output dtypes are identical.
WARNING:tensorflow:The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
W0816 20:24:48.960490 140309352765248 image_ops_impl.py:2018] The operation `tf.image.convert_image_dtype` will be skippedsince the input and output dtypes are identical.
WARNING:tensorflow:The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
W0816 20:24:48.966012 140309352765248 image_ops_impl.py:2018] The operation `tf.image.convert_image_dtype` will be skippedsince the input and output dtypes are identical.
WARNING:tensorflow:The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
W0816 20:24:48.971490 140309352765248 image_ops_impl.py:2018] The operation `tf.image.convert_image_dtype` will be skippedsince the input and output dtypes are identical.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04.4 (Nvidia NGC Tensorflow 20.06 docker image)
• Mobile device name if the issue happens on a mobile device: NA
• TensorFlow installed from (source or binary): source (Nvidia NGC Tensorflow 20.06 docker image)
• TensorFlow version (use command below): 2.2.0
• Python version: 3.6
• Bazel version (if compiling from source): NA (Nvidia NGC Tensorflow 20.06 docker image)
• GCC/Compiler version (if compiling from source): NA (Nvidia NGC Tensorflow 20.06 docker image)
• CUDA/cuDNN version: CUDA 11.0 / cuDNN 8.0.1.13
• GPU model and memory: GeForce GTX1080 8GB