Auto3Dseg cuda OOM during Ensembling

Describe the bug
Models have all finished training, and during the ensembling process, cuda runs out of memory.

Reproduce
Steps to reproduce the behavior:
Run Autorunner on AMOS22 dataset

manually resetting cuda cache, restarting kernel and instance all come back to this error.

Expected behavior
training proceeds without error
MONAI version: 1.2.0
Numpy version: 1.25.2
Pytorch version: 2.0.1+cu117
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: c33f1ba588ee00229a309000e888f9817b4f1934
MONAI file: /home/exouser/.local/lib/python3.10/site-packages/monai/init.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.3.0
Nibabel version: 5.1.0
scikit-image version: 0.21.0
Pillow version: 9.0.1
Tensorboard version: 2.14.0
gdown version: 4.7.1
TorchVision version: 0.15.2+cu117
tqdm version: 4.66.1
lmdb version: 1.4.1
psutil version: 5.9.0
pandas version: 2.0.3
einops version: 0.6.1
transformers version: 4.21.3
mlflow version: 2.6.0
pynrrd version: 1.0.0

Environment (please complete the following information):
OS: ubuntu 22.04
Python 3.10.12
Driver Version: 525.85.05 CUDA Version: 12.0
GRID A100X-40C - 125GB RAM

I'm happy to provide any other logs to help, this is the second time I've run into this issue, the issue persists after a full kernel restart and RAM clearing.

Inizia con il percorso di ensembling di Autorunner e riproduci il problema sul dataset AMOS22 utilizzando le versioni e l’ambiente elencati nell’issue. Non è indicato alcun file sorgente o test; il lavoro è completato quando l’ensembling termina senza un errore CUDA out-of-memory e il training procede correttamente.