ageitgey / ageitgey/face_recognition

GPU memory not get released after using CNN for face location in web_service_example.py (flask example)

Open
#722 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
56.8k
Forks
13.7k
PR merge metrics
No merged PRs in 30d

Description

  • face_recognition version: 1.2.3
  • Python version: 3.5.2
  • Operating System: Ubuntu 16.04 LTS
  • GPU: NVIDIA GeForce GTX 1050TI SSE2
Description

GPU memory not released after adding "CNN for face location" line in web_service_example.py

What I Did
By adding **line of finding face location using CNN** in web_service_example.py like this ::
    **# Load the uploaded image file
    img = face_recognition.load_image_file(file_stream)

    _# Get face locations for any face in the uploaded image
    unknown_face_location = face_recognition.face_locations(img, number_of_times_to_upsample=1, model="cnn")_

    # Get face encodings for any faces in the uploaded image
    unknown_face_encodings = face_recognition.face_encodings(img, unknown_face_location)**

i found that my GPU does not release memory after CNN,
So when i try to upload image 1st time, it uses 77% of GPU's memory and when i upload image 2nd time it starts using GPU' memory from 77% ( that means GPU is not releasing its memory after complition of request),
after multiple attempts GPU's memory reach to 100% and then it shows ::
  **return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudaMalloc(&data, n) in file /home/dlib-19.13/dlib/cuda/cuda_data_ptr.cpp:28. code: 2, reason: out of memory**


i tried ::
**del unknown_face_location
gc_collect()**
below unknown_face_location
but it didn't work, 
when i restart flask only then GPU releases memory (which is not good option to use)

i used following version::
Nvidia Driver : nvidia-396
CUDA Version 9.0.176
Cudnn Version 7.4.2
Dlib Version 19.13

I need to know how to overcome from this memory issue by using another drivers and version or by changing the code.
NOTE: i need CNN and when i use CNN in a code where i don't use Flask and other web service applications it works fine.(I am new to this So spare me for any small mistakes done by me)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with web_service_example.py and reproduce repeated Flask uploads using face_recognition.face_locations(..., model="cnn"). Inspect the request lifecycle and the CNN detector's GPU allocation behavior with the listed Python, dlib, CUDA, and cuDNN versions. Done means repeated requests no longer grow GPU usage until cudaMalloc fails.

Written by the indexing model from the issue text.

Assessment

Tech stack
flask, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.