Single instance parallel threads - Retrieval/Preview & Image processing

Describe the feature you'd like to request

Parallel processing of tasks within single run - currently it is sequential processing.

Describe the solution you'd like

Currently Recognize goes in order:

One image type processing:

• preview generation/retrieval
• NN processing
• pushing out results
  Second image type processing:
• preview generation/retrieval
• NN processing
• pushing out results

Because of GPU memory limitations two Recognize instances can not be run in parallel as it will cause errors on GPU side memory exhaustion (OOM) and bad results/images being skipped.

Therefore the ask is to have job queues by type:

• preview generation/retrieval putting pictures into the queue for processing, and once one type of images (Object, Landmar, Face) is finished moving to another, as Network/Disk I/O takes time too,
• NN processing would pick up images to process as are ready and finished from the image type queue (it needs to be by type, as launching Node with tensorflow takes couple of seconds too). Results could be put to another queue by type of image.
• pushing out results - similar to above it would pick up tasks to be performed based from separate queue.

What is expected difference?
Currently with sequential approach, GPU is in sleep mode whilst retrieval or pushing out of results is being done (not sure about the latter as these seem to be currently merged). Retrieval takes similar time to processing at least in my situation.

This means that with parallel processing speedup could be almost of 200% as all resources would be utilized in parallel.

Describe alternatives you've considered

Running multiple Recognize instances causes GPU memory exhaustion/oom and causes images to be skipped.

No implementation files or tests are named. Start by tracing the current sequential flow for preview/retrieval, NN processing, and result output; done means these stages overlap through per-image-type queues without GPU out-of-memory failures or skipped images.