Unable to reproduce faster rcnn and retinanet r50 fpn 1x

1. Full runnable code or full changes you made:
```
I simply clone the full repository and do not make any changes. And I have tried both the latest 0.6 version and a previous verion of 0.5.
```

2. What exact command you run:
```
python tools/train_net.py --config-file configs/COCO-Detection/faster_rcnn_R_50_FPN_1x.yaml --num-gpus 8
```

3. __Full logs__ or other relevant observations:
```
[02/17 11:02:39 d2.engine.hooks]: Overall training speed: 1015 iterations in 0:07:15 (0.4287 s / it)
[02/17 11:02:39 d2.engine.hooks]: Total training time: 0:07:15 (0:00:00 on hooks)
[02/17 11:02:39 d2.utils.events]: eta: 10:35:48 iter: 1017 total_loss: 1.486 loss_cls: 0.5452 loss_box_reg: 0.2228 loss_rpn_cls: 0.4857 loss_rpn_loc: 0.2562 time: 0.4286 data_time: 0.0278 lr: 0.02 max_mem: 2895M
Traceback (most recent call last):
File "tools/train_net.py", line 169, in
args=(args,),
File "/home/ssd2/xx/detectron2-0.5/detectron2/engine/launch.py", line 79, in launch
daemon=False,
File "/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/ssd2/wangyu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
while not context.join():
File "/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 150, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/home/ssd2/xx/detectron2-0.5/detectron2/engine/launch.py", line 125, in _distributed_worker
main_func(*args)
File "/home/ssd2/xx/detectron2-0.5/tools/train_net.py", line 157, in main
return trainer.train()
File "/home/ssd2/xx/detectron2-0.5/detectron2/engine/defaults.py", line 489, in train
super().train(self.start_iter, self.max_iter)
File "/home/ssd2/xx/detectron2-0.5/detectron2/engine/train_loop.py", line 149, in train
self.run_step()
File "/home/ssd2/xx/detectron2-0.5/detectron2/engine/defaults.py", line 499, in run_step
self._trainer.run_step()
File "/home/ssd2/xx/detectron2-0.5/detectron2/engine/train_loop.py", line 273, in run_step
loss_dict = self.model(data)
File "/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ssd2/xx/detectron2-0.5/detectron2/modeling/meta_arch/rcnn.py", line 157, in forward
proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
File "/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ssd2/xx/detectron2-0.5/detectron2/modeling/proposal_generator/rpn.py", line 478, in forward
anchors, pred_objectness_logits, pred_anchor_deltas, images.image_sizes
File "/home/ssd2/xx/detectron2-0.5/detectron2/modeling/proposal_generator/rpn.py", line 511, in predict_proposals
self.training,
File "/home/ssd2/xx/detectron2-0.5/detectron2/modeling/proposal_generator/proposal_utils.py", line 104, in find_top_rpn_proposals
"Predicted boxes or scores contain Inf/NaN. Training has diverged."
FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.
```

## Expected behavior:
The performance of retinanet r50 fpn 1x in model zoo is 37.4 while the model I reproduce is 34.1. And the training of faster rcnn diverges after a few iterations.

## Environment:

Paste the output of the following command:
```
---------------------- ---------------------------------------------------------------------------------
sys.platform linux
Python 3.7.0 (default, Jun 28 2018, 13:15:42) [GCC 7.2.0]
numpy 1.21.6
detectron2 0.5 @/home/ssd2/xx/detectron2-0.5/detectron2
Compiler GCC 5.4
CUDA compiler CUDA 10.2
detectron2 arch flags 6.1
DETECTRON2_ENV_MODULE
PyTorch 1.10.2+cu102 @/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torch
PyTorch debug build False
GPU available Yes
GPU 0,1,2,3,4,5,6,7 Tesla P40 (arch=6.1)
CUDA_HOME /usr/local/cuda
Pillow 9.2.0
torchvision 0.11.3+cu102 @/home/ssd2/xx/anaconda3/lib/python3.7/site-packages/torchvision
torchvision arch flags 3.5, 5.0, 6.0, 7.0, 7.5
fvcore 0.1.5.post20221221
iopath 0.1.9
cv2 4.6.0
---------------------- ---------------------------------------------------------------------------------
PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 10.2
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
- CuDNN 7.6.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
```