CUDA Memory Consumption

I've been noticing a fairly high memory consumption when running training with Detectron2. In particular, I have the following setup:

**Graphics card:** GeForce RTX 2080 Ti
**Model:** Faster-RCNN R50-FPN 3x
**Image res:** 1382 x 512 (KITTI standard)
**Training dataset size:** 1500
**Batch size:** 8

And this is my setup for the cfg file, closely following the Colab tutorial:

```python
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file(model_path))
cfg.DATASETS.TRAIN = train_datasets
cfg.DATASETS.TEST = test_datasets
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.DEVICE = "cuda"
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(model_path) # Get initial weights from model zoo
cfg.SOLVER.IMS_PER_BATCH = 8 # Batch size
cfg.SOLVER.BASE_LR = 0.00025 # Learning rate
cfg.SOLVER.MAX_ITER = 500 # Max number of iterations to train for
cfg.SOLVER.STEPS = [] # Do not decay learning rate
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1
```

Now, running `nvidia-smi` during training yields the following:
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:18:00.0 Off | N/A |
| 47% 78C P2 176W / 250W | 8667MiB / 11018MiB | 80% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 258766 C /usr/NX/bin/nxnode.bin 167MiB |
| 0 N/A N/A 2822702 C python 8485MiB |
+-----------------------------------------------------------------------------+

```
However, from the training output itself, `max_mem` is shown to be only about 6GB. I was wondering whether this level of consumption was normal? Because of this, I'm constrained in increasing the batch size; since I'll otherwise end up with a runtime error that CUDA has run out of memory. I was initially working with a larger image resolution, but the results seem to be about the same regardless.

If there are any suggestions for how I could fix this, that would be greatly appreciated. And if there is any further clarification I can provide, feel free to let me know.

Thanks in advance!