Memory Issues on applications with large collections upon restarting docker

**Description**

java.lang.OutOfMemoryError: Java heap space

I have compreface application with few hundred thousand subjects. which works normally flawless. But if I attempt to restart or down and up the docker. Those large collection applications fails to provide recognition (Error details attached). However I am still able to add more subjects. And the applications with smaller collection have no issues recognizing faces.

I used to have 2 processes and 2 threads. Each process occupies almost 4GB of my GPU memory. GPU has 12GB available memory (RTX 3060 12GB)

**Background**
I have built a system which stores every face captured in some public cameras into an FTP folder. Each image is just an inch small sized about 12 KB - 25 KB. A script runs continuously to add those faces into an application.
Upon reaching about 700,000 faces. It is working pretty fast. The recognition service is utilized only upon request. i.e. the user attempt to search someone by image. the system tells the occurrences of that person by running recognition service in that application.
I found that the Swap memory was full, so I restarted the docker. after that, the docker failed to provide search (recognition) in that application. I tried restarting several times but still got the issue. However the issue resolved automatically after few hours.
Note: i have another application with very small collection i.e. 10-15 subjects only. No issues on searching in that application. which concludes that the docker is running fine.
I tried 3 times restarting docker when it was running normally. Each time. it went into that error state where it is unable to recognize faces in the large collection.
I checked the logs. In the long details, I found this text "**Caused by: java.lang.OutOfMemoryError: Java heap space**".
So I split the subjects into two different applications by manually running query to update API_KEY in the subject table. Issue still persisted.
I have a lot of RAM available. I don't know where i am making mistake. The search is fast. The RAM is available. I am wondering if the subjects are loaded into GPU RAM which is only 8 GB then it might be the issue. But if it is loaded into System RAM which is 64GB and a small portion is utilized only. Rest is free.

Here is my docker-compose.yml file:

yns@faceserver:/var/CompreFace_1.1.0/custom-builds/SubCenter-ArcFace-r100-gpu$ cat docker-compose.yml
version: '3.4'

volumes:
postgres-data:

services:
compreface-postgres-db:
image: ${registry}compreface-postgres-db:${POSTGRES_VERSION}
restart: always
container_name: "compreface-postgres-db"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_DB=${postgres_db}
volumes:
- postgres-data:/var/lib/postgresql/data

compreface-admin:
image: ${registry}compreface-admin:${ADMIN_VERSION}
restart: always
container_name: "compreface-admin"
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- ENABLE_EMAIL_SERVER=${enable_email_server}
- EMAIL_HOST=${email_host}
- EMAIL_USERNAME=${email_username}
- EMAIL_FROM=${email_from}
- EMAIL_PASSWORD=${email_password}
- ADMIN_JAVA_OPTS=${compreface_admin_java_options}
- MAX_FILE_SIZE=${max_file_size}
- MAX_REQUEST_SIZE=${max_request_size}B
depends_on:
- compreface-postgres-db
- compreface-api

compreface-api:
image: ${registry}compreface-api:${API_VERSION}
restart: always
container_name: "compreface-api"
depends_on:
- compreface-postgres-db
environment:
- POSTGRES_USER=${postgres_username}
- POSTGRES_PASSWORD=${postgres_password}
- POSTGRES_URL=jdbc:postgresql://${postgres_domain}:${postgres_port}/${postgres_db}
- SPRING_PROFILES_ACTIVE=dev
- API_JAVA_OPTS=${compreface_api_java_options}
- SAVE_IMAGES_TO_DB=${save_images_to_db}
- MAX_FILE_SIZE=${max_file_size}
- MAX_REQUEST_SIZE=${max_request_size}B
- CONNECTION_TIMEOUT=${connection_timeout:-10000}
- READ_TIMEOUT=${read_timeout:-60000}

compreface-fe:
image: ${registry}compreface-fe:${FE_VERSION}
restart: always
container_name: "compreface-ui"
ports:
- "8000:80"
depends_on:
- compreface-api
- compreface-admin
environment:
- CLIENT_MAX_BODY_SIZE=${max_request_size}
- PROXY_READ_TIMEOUT=${read_timeout:-60000}ms
- PROXY_CONNECT_TIMEOUT=${connection_timeout:-10000}ms

compreface-core:
image: ${registry}compreface-core:${CORE_VERSION}
restart: always
container_name: "compreface-core"
runtime: nvidia
environment:
- ML_PORT=3000
- UWSGI_PROCESSES=${uwsgi_processes:-2}
- UWSGI_THREADS=${uwsgi_threads:-1}

-----------------------------------------------------------

In the .env file. I set 2 processes and 2 threads. So GPU with 12GB memory is used around 8GB.
Also tried with 1 thread.

**To Reproduce**

Steps to reproduce the behavior:
Add few hundred thousand subjects into a single recognition service.
Try to search (recognize) into that collection. It will run fine.
Then run:
- sudo docker compose down
- sudo docker compose up -d

I may go into that error state. watch the logs by not running docker in detached mode.

**Screenshots**


Memory seems to be available in docker stats. Here is the screenshot:

**Desktop:**

- OS: Ubuntu Server 22.04
- CPU: i7-13700k
- Memory: Corsair Vengeance 64GB(16x4) DDR-5 5600 MHz
- Mainboard: Gigabyte Z790
- Disk: Samsung 980 Pro 2TB Gen-4 NVME
- GPU: Nvidia RTX 3060 (12GB)

**Another log error I found:**
I found another issue during testing and restarting etc. which was different from out of memory.


**Additional context**

Please guide what to do. or is it some bug to be fixed in the next version?
The out of memroy issue is strange. i can see in the docker stats. all 62GB is assigned to the containers. Is it storing data in the main RAM of the GPU RAM. If GPU RAM. Then the reason is obvious. I will use 3090 then. which has 24GB memory
And if it is using System RAM. I will upgrade to 128 GB or higher. but that doesn't seem to be the problem. i sill have free RAM availble in the system monitoring.