LFW accuracy evaluation and usage suggestions

• face_recognition version: 1.2.3
• Python version: 3.6 anaconda
• Operating System: Windows

Description

I'm trying to measure the accuracy of this face recognition code and apparently I'm doing it wrong, as I'm getting values around 85% which is quite far from 99% or 97% as reported by some other users. Although I'm probably not following the protocol defined by LFW, I don't think any protocol would get me from 85% to 99%, so there must be something wrong with how I use the code. I would appreciate any help understanding what I am doing wrong.

This is a very practical question, as I cannot get the best performance out of the code if I don't know how to use it in the right way.

What I Did

I'm following this tutorial, except that I use face_recognition instead of the eigenfaces as described here:
https://scikit-learn.org/stable/auto_examples/applications/plot_face_recognition.html

For face_recognition, I first compute and store encodings for every train image (at least 1 for every person), and then compute encodings for every test image and find the closest encoding in the training set, which determines the output label. I also discard any data points in training and test sets where the faces were not recognized at all (usually about 6%), as not doing so would bring down the accuracy even further. I also tried converting RGB images to BGR (in case the models were trained on cv2 BGR images), but that also didn't help.

My thoughts on what could be wrong, but I would also like to hear what you think:

1. perhaps I need to find the average distance of each test sample to all train samples of a particular class and then find the minimum across all classes instead of just finding the closest train sample? Or maybe some other metric, e.g. look for the 3 closest samples in each class and then take the average?
2. min_faces_per_person is 4 and test_size is 0.75 which means that there could be classes with 1 labeled face and 3 test faces. Maybe that's too low and I need at least several test faces?
3. the sklearn images are RGB which means that they were probably loaded with skimage/imageio. If the face_recognition model was trained with cv2 loaded BGR images, using skimage/imageio images may degrade the accuracy (even if I swap the dimensions to convert RGB -> BGR). Turns out different libraries read jpgs differently and I've seen reports where switching the image reading library impacts the model performance: https://github.com/scikit-image/scikit-image/issues/2293#issuecomment-345445933
   Maybe I need to obtain the original LFW images which were used for training and read them using cv2?
4. the results are actually fine and somehow LFW evaluation metric is way more lenient than classification_report and would report actual 99% on the similar output. In that case I would still like to know whether you think I should try 1-3 to stick with best performance.

My code is as follows:

import logging
from time import time

import cv2
import face_recognition
import numpy as np
from sklearn.datasets import fetch_lfw_people
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split

logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')

ENCODING_DIM = 128

###############################################################################
# Download the data, if not already on disk and load it as numpy arrays

lfw_people = fetch_lfw_people(min_faces_per_person=4, color=True, resize=1)

# introspect the images arrays to find the shapes (for plotting)
n_samples, h, w, _ = lfw_people.images.shape

X = lfw_people.images
n_features = X.shape[1]

# the label to predict is the id of the person
y = lfw_people.target
target_names = lfw_people.target_names
n_classes = target_names.shape[0]

print("Total dataset size:")
print("n_samples: %d" % n_samples)
print("n_features: %d" % n_features)
print("n_classes: %d" % n_classes)

# split into a training and testing set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.75)

n_test_samples = X_test.shape[0]
n_train_samples = X_train.shape[0]

t0 = time()

# find encodings for each training image
train_encodings = np.zeros((n_train_samples, ENCODING_DIM))
missing_indexes = []
for i in range(n_train_samples):
    print('Finding train encodings for ', i)
    image = X_train[i, :, :, :].astype(np.uint8)
    # image = X_train[i, :, :, :].astype(np.uint8)[:, :, [2, 1, 0]]  # permutations to convert skimage to cv2 image
    encoding = face_recognition.face_encodings(image)
    if encoding:
        train_encodings[i, :] = encoding[0].reshape(1, -1)
    else:
        missing_indexes.append(i)

# Remove training examples where face was not detected and encodings are unavailable
X_train = np.delete(X_train, missing_indexes, axis=0)
y_train = np.delete(y_train, missing_indexes, axis=0)
train_encodings = np.delete(train_encodings, missing_indexes, axis=0)

# find encodings for each test image and find closest L2-distance encoding in the training set
y_pred = np.zeros((n_test_samples, 1), dtype=np.int32)
missing_test_indexes = []
for i in range(n_test_samples):
    print('Finding closest face for ', i)
    # image = X_test[i, :, :, :].astype(np.uint8)[:, :, [2, 1, 0]]  # permutations to convert skimage to cv2 image
    image = X_test[i, :, :, :].astype(np.uint8)
    # cv2.imwrite("C:\\Users\\eternity\\test\\lfw\\{0}.jpg".format(str(i)), image)  # check the test images
    encoding = face_recognition.face_encodings(image)
    if encoding:
        match_index = np.linalg.norm(train_encodings - encoding[0], axis=1).argmin()
        y_pred[i] = y_train[match_index]
    else:
        # Not found
        y_pred[i] = -1
        missing_test_indexes.append(i)

print("Could not detect {0}/{1} train faces".format(len(missing_indexes), n_train_samples))
print("Could not detect {0}/{1} test faces".format(np.sum(y_pred == -1), n_test_samples))

# Remove test examples where face was not detected and comparison could not be made
y_pred = np.delete(y_pred, missing_test_indexes, axis=0)
y_test = np.delete(y_test, missing_test_indexes, axis=0)

print("done in %0.3fs" % (time() - t0))

labels = list(range(n_classes))
target_names = list(target_names)
# labels = [-1] + list(range(n_classes))
# target_names = ['Not found'] + list(target_names)

print(classification_report(y_test, y_pred, target_names=target_names, labels=labels))
print(confusion_matrix(y_test, y_pred, labels=labels))

# Accuracy
#
# No permutations (RGB):
#
# Could not detect 100/1683 train faces
# Could not detect 292/5050 test faces
#
#                      micro avg       0.88      0.88      0.88      4758
#                      macro avg       0.69      0.78      0.72      4758
#                   weighted avg       0.80      0.88      0.83      4758
#
# With permutations (BGR):
#
# Could not detect 92/1683 train faces
# Could not detect 300/5050 test faces
#
#                      micro avg       0.85      0.85      0.85      4750
#                      macro avg       0.68      0.76      0.70      4750
#                   weighted avg       0.78      0.85      0.81      4750

Start by reproducing the provided Python script, focusing on fetch_lfw_people, train_test_split, and face_recognition.face_encodings. The payload names no project files or tests, so first compare the evaluation procedure with the LFW protocol and determine whether the reported accuracy difference is expected. Done means documenting the cause and a reproducible evaluation procedure.