TensorFlow object detection API evaluate training performance on both training data and validation data

System information

• What is the top-level directory of the model you are using: object detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.3 LTS
• TensorFlow installed from (source or binary): from source
• TensorFlow version (use command below): 'v1.12.2-0-g6b634657d8' 1.12.2
• Bazel version (if compiling from source): 0.17.2
• CUDA/cuDNN version: cuda 10.0 / cuDnn 7.3
• GPU model and memory: GeForce RTX 2080 Ti Rev. A (11 GB)
• Exact command to reproduce:

python model_main.py --alsologtostderr \
  --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
  --model_dir=${TRAIN_DIR} \
  --num_train_steps=24000 \
  --sample_1_of_n_eval_on_train_examples=25 \
  --num_eval_steps=100 \
  --sample_1_of_n_eval_examples=1 \
  --eval_training_data=True

Describe the problem

To identify overfitting, I need both performances on training data and validation data. I am using my data. And my mask_rcnn_resnet101_atrous_coco NN is not performing well on the validation dataset. So, I wanted to know the performance of NN on training data and validation data during a training session.

I found similar problems on StackOverflow but no solution.

1. how to check both training/eval performances in TensorFlow object_detection
   
2. How to calculate evaluation metrics on training data in TensorFlow's Object Detection API?
   
   And I also post a question TensorFlow object detection API evaluate training performance
   

I used --eval_training_data=True as a parameter but the run performed worse than run without it. However, --eval_training_data=True suppose to run evaluation on the training dataset and should have a better precision (mAP) result than the validation dataset. I could not find any option to run the calculation on both datasets at the same time and report them separately during a training session.

Source code / logs

Evaluation result with --eval_training_data=True included:

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.165
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.281
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.167
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.051
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.109
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.202
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.164
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.202
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.202
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.141
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.236

Evaluation result without --eval_training_data :

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.168
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.283
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.173
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.049
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.108
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.208
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.170
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.208
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.208
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.056
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.139
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.248

How to run the script to compute accuracy (mAP) both on the validation set and the training set (randomly some percentage) and report them separately during a training session?