"an illegal memory access was encountered"

**Describe the bug**

Hi ! Just switched from MobileNet-GPU to ArcFace-GPU, and I'm running into a memory issue while doing facial identification.
I'm currently running on a GTX 970 (Driver version: 510.108.03) with a single process (Got 4GB of ram, model uses ~3GB).
Everything works fine, but the gpu ram usage builds up slowly, even through requests to add new faces are queued client-side with a max concurrency of 5.
It goes from initially ~3GB to ~3.3GB then memory goes back ~3.1GB (Garbage collector ?) but then every request return the following error:

`Error during synchronization between servers: [500 INTERNAL SERVER ERROR] during [POST] to [http://compreface-core:3000/find_faces] [FacesFeignClient#findFaces(MultipartFile,Integer,Double,String)]: [{"message":"MXNetError: Traceback (most recent call last):\n File \"../src/storage/./pooled_storage_manager.h\", line 96\nCUDA: an illegal memory access was encountered"} ] `

**To Reproduce**

Launch ArcFace GPU with only 1 process. (With a ~4GB GPU to reproduce the conditions)
Add faces 5 by 5.
Wait a few minutes (~10 minutes in my last test)

**Expected behavior**

Memory usage shouldn't grow over time and previously processed images shouldn't mess up with the GPU's memory

**Setup:**

- OS:Ubuntu 22.04
- GPU: GTX 970
- Driver: 510.108.03
- Docker: 20.10.23 (Build 7155243)

Thanks for reading and for your help !