Issues in running face_recognition and TensorFlow together in Flask server

• face_recognition version: 1.2.3
• Python version: 3.5.5
• Operating System: Windows 10

Description

I'm trying to build a recognition service, that can both identify known faces and everyday objects from a directory of images. I'm using Flask and Waitress to deploy a simple server that uses your library to perform face recognition and some models from TensorFlow Model Zoo to perform object recognition. I have both tasks separated into 2 requests: /face and /object. You give a directory to the server and it will analyze all the images in that directory. The reason I'm using a web server is that I need to have the TensorFlow models loaded in memory all the time because they take a lot of time to load and I need quick predictions.

The issue here is that when TensorFlow responds to an object recognition request, whenever I start a face recognition request, or vice-versa, the process stops and waits until the other task is completed. I checked if the server was running in multithreaded mode and it was. I don't know if this is an issue with CUDA (I compiled both TensorFlow and dlib with CUDA support) or if it is a problem with the way I was doing things.

What I Did

I analyze the images in a directory one by one, so I tried to introduce a reentrant lock whenever it starts identifying faces/objects in one image. I thought that with this I could alternate between both requests. When analysing a directory with 77 frames extracted from a video, checking the following logs you can see that the object recognition request ended first and the face recognition request has stopped and only started working when the object recognition stopped.

• Single face recognition request
  [2019-02-14 10:12:07.955267] [predict_service::predict_dir_face] REQUEST 82fbc622-86d2-4499-9704-7e1407531252: POST /face
[2019-02-14 10:13:52.207176] [predict_service::predict_dir_face] REQUEST 82fbc622-86d2-4499-9704-7e1407531252: EXEC_TIME (sec): 100.16752982139587

• Single object recognition request
  [2019-02-14 10:21:19.903644] [predict_service::predict_dir_object] REQUEST 5a156847-64fc-4a36-8c7b-159d045b97e6: POST /object
[2019-02-14 10:25:30.385987] [predict_service::predict_dir_object] REQUEST 5a156847-64fc-4a36-8c7b-159d045b97e6: EXEC_TIME (sec): 225.4509048461914

• Both requests in parallel
  [2019-02-14 10:26:25.113477] [predict_service::predict_dir_face] REQUEST 63677299-1420-4be3-966d-66e0cf4fb6b1: POST /face
[2019-02-14 10:26:32.182800] [predict_service::predict_dir_object] REQUEST a7a6cd15-f824-4674-bb10-b60993bc4e03: POST /object
[2019-02-14 10:30:27.626746] [predict_service::predict_dir_object] REQUEST a7a6cd15-f824-4674-bb10-b60993bc4e03: EXEC_TIME (sec): 235.44491577148438
[2019-02-14 10:31:49.794100] [predict_service::predict_dir_face] REQUEST 63677299-1420-4be3-966d-66e0cf4fb6b1: EXEC_TIME (sec): 324.68062376976013

Start with the Flask/Waitress handlers for the /face and /object requests and reproduce the parallel workload using the directory of 77 extracted frames. Compare behavior with the TensorFlow and dlib CUDA configurations described in the report; done means identifying whether the blocking is caused by the server execution model or shared compute resources and documenting the supported setup.