Bug in mAP/mAR calculation for reference object detection tutorial

🐛 Bug

I believe I've found a bug in the mAP/mAR evaluation code for the TorchVision Object Detection Finetuning Tutorial.

To Reproduce

If you run the Jupyter Notebook version of the tutorial on Colab and make any of the following modifications, you can reproduce this bug.

1. In the training loop, add evaluate(model, data_loader, device=device) to evaluate performance on the training set, or
2. When defining dataset_test, use get_transform(train=True) instead of train=False, or
3. Always add RandomHorizontalFlip in get_transform

Expected behavior

Whether or not RandomHorizontalFlip is added shouldn't have a substantial impact on mAP/mAR performance. However, I've noticed that adding RandomHorizontalFlip results in significantly lower mAP/mAR scores (~0.2 instead of ~0.7).

Environment

This is the default environment on Google Colab:

PyTorch version: 1.8.1+cu101
Is debug build: False
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.0.221
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 460.32.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.8.1+cu101
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.9.1
[pip3] torchvision==0.9.1+cu101
[conda] Could not collect

Reproduce the issue in the Jupyter Notebook version of the TorchVision Object Detection Finetuning Tutorial on Colab. Compare the training-loop evaluate call, dataset_test transforms, and RandomHorizontalFlip cases, then verify that the mAP/mAR results no longer drop substantially between configurations.