Monai Label holds on to GPU memory when facing OOM error

Describe the bug
When running inference of totalSegmentator with large volumes, I do get OOM error. I was expecting I can change the sw_device to use CPU and rerun since I added a config var to the inference. However Monai label still holds on to the gpu memory so No one can rerun or use that memory unless the server is stopped and restarted

Server logs

2023-04-10 22:11:29,405] [16464] [INFER_0] [INFO] (monailabel.tasks.infer.basic_infer:412) - Infer model path: /MONAILabel/sample-apps/radiology/model/segmentation.pt
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 224/224 [00:18<00:00, 12.09it/s]
[2023-04-10 22:11:48,654] [16464] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/conda/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/opt/conda/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/opt/conda/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/conda/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/lib/python3.8/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/opt/conda/lib/python3.8/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/MONAILabel/monailabel/endpoints/infer.py", line 179, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/MONAILabel/monailabel/endpoints/infer.py", line 161, in run_inference
    result = instance.infer(request)
  File "/MONAILabel/monailabel/interfaces/app.py", line 298, in infer
    result_file_name, result_json = f.result(request.get("timeout", settings.MONAI_LABEL_INFER_TIMEOUT))
  File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/MONAILabel/monailabel/interfaces/app.py", line 295, in run_infer_in_thread
    return t(r)
  File "/MONAILabel/monailabel/tasks/infer/basic_infer.py", line 299, in __call__
    data = self.run_inferer(data, device=device)
  File "/MONAILabel/monailabel/tasks/infer/basic_infer.py", line 473, in run_inferer
    outputs = inferer(inputs, network)
  File "/opt/monai/monai/inferers/inferer.py", line 202, in __call__
    return sliding_window_inference(
  File "/opt/monai/monai/inferers/utils.py", line 252, in sliding_window_inference
    output_image_list[ss] = (output_image_list[ss] / count_map_list.pop(0)).to(compute_dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 21.84 GiB (GPU 0; 31.75 GiB total capacity; 23.38 GiB already allocated; 6.73 GiB free; 23.39 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Expected behavior
There should be a try Except block or something to release the gpu memory after a failed inference. In real deployment systems there would be cases where inference would fail as OOM or bad data. System should always free up memory unless we want to keep the model hot and always loaded

Start by reading monailabel/tasks/infer/basic_infer.py around run_inferer and monailabel/interfaces/app.py around infer and run_infer_in_thread, using the traceback as the entry point. Reproduce the CUDA out-of-memory path with the reported inference setup, then verify that a failed inference releases usable GPU memory and a subsequent inference can run without restarting the server.