Not enough memory on RTX 3090 to train ViTDet?

If you do not know the root cause of the problem, please post according to this template:

## Instructions To Reproduce the Issue:

I'm trying to train an instance segmentation ViTDet model with a custom and relatively small dataset (6000 images of 640x480)
I'm using Windows 10 and RTX 3090.
I'm trying to train using the basic configuration ViTDet, ViT-B), which in theory, should take 12.3gb. My RTX 3090 has 24gb and I get a CUDA out of memory message.
I reduced the batch size from 64 to 2, reduced the number of workers to 2 and used FP16 comperssion, none of it solved the issue.
I checked the config.yaml in output and I see that these are my actual settings.

here's my environment:

sys.platform win32
Python 3.10.5 | packaged by conda-forge | (main, Jun 14 2022, 06:57:19) [MSC v.1929 64 bit (AMD64)]
numpy 1.23.1
detectron2 0.6 @C:\detectron2\detectron2
Compiler MSVC 193231332
CUDA compiler CUDA 11.6
detectron2 arch flags C:\detectron2\detectron2\_C.cp310-win_amd64.pyd; cannot find cuobjdump
DETECTRON2_ENV_MODULE
PyTorch 1.12.0 @C:\cuda\miniconda\envs\mmopenlab\lib\site-packages\torch
PyTorch debug build False
GPU available Yes
GPU 0,1 NVIDIA GeForce RTX 3090 (arch=8.6)
Driver version 516.59
CUDA_HOME C:\cuda\116
Pillow 9.2.0
torchvision 0.13.0 @C:\cuda\miniconda\envs\mmopenlab\lib\site-packages\torchvision
torchvision arch flags C:\cuda\miniconda\envs\mmopenlab\lib\site-packages\torchvision\_C.pyd; cannot find cuobjdump
fvcore 0.1.5.post20220512
iopath 0.1.9
cv2 4.6.0
---------------------- --------------------------------------------------------------------------------------------
PyTorch built with:
- C++ Version: 199711
- MSVC 192829337
- Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
- OpenMP 2019
- LAPACK is enabled (usually provided by MKL)
- CPU capability usage: AVX2
- CUDA Runtime 11.6
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
- CuDNN 8.3.2 (built against CUDA 11.5)
- Magma 2.5.4
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -IC:/cb/pytorch_1000000000000/work/mkl/include -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,