deepinsight / deepinsight/insightface

Why the evaluation result on IJB-B is different from paper&your test(almost same code)?

Open
#2,242 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

First of all, thank you for your excellent works, I learn a lot from your repository.
here is my puzzle:
my code is :
```
import os
import matplotlib
matplotlib.use('Agg')
from pathlib import Path
import pandas as pd
import numpy as np
from prettytable import PrettyTable

import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc
from menpo.visualize.viewmatplotlib import sample_colours_from_colourmap

def read_template_pair_list(path):
# pairs = np.loadtxt(path, dtype=str)
pairs = pd.read_csv(path, sep=' ', header=None).values
# print(pairs.shape)
# print(pairs[:, 0].astype(np.int))
t1 = pairs[:, 0].astype(np.int)
t2 = pairs[:, 1].astype(np.int)
label = pairs[:, 2].astype(np.int)
return t1, t2, label

if __name__ == "__main__":

score_save_file = r"I:\Dataset\IJB_release\IJBB\result\MS1MV2-ResNet100-ArcFace-TestMode(N1D1F1).npy"
#MS1MV2-ResNet100-Arcface_IJBB_N1D1F1

image_path = "./IJBB"
pdf_name = "MS1MV2-ResNet100-ArcFace"
target = "IJBB"
p1, p2, label = read_template_pair_list(
os.path.join(r"I:\Dataset\IJB_release\IJBB\meta", #'%s/meta' % image_path,
'%s_template_pair_label.txt' % target.lower()))

files = [score_save_file]
methods = []
scores = []
for file in files:
methods.append(Path(file).stem)
scores.append(np.load(file))

methods = np.array(methods)
scores = dict(zip(methods, scores))
colours = dict(
zip(methods, sample_colours_from_colourmap(methods.shape[0], 'Set2')))
x_labels = [10 ** -6, 10 ** -5, 10 ** -4, 10 ** -3, 10 ** -2, 10 ** -1]
#x_labels = [10 ** -5, 10 ** -4, 10 ** -3, 10 ** -2, 10 ** -1]
tpr_fpr_table = PrettyTable(['Methods'] + [str(x) for x in x_labels])
fig = plt.figure()
for method in methods:
fpr, tpr, _ = roc_curve(label, scores[method])
roc_auc = auc(fpr, tpr)
fpr = np.flipud(fpr)
tpr = np.flipud(tpr) # select largest tpr at same fpr
plt.plot(fpr,
tpr,
color=colours[method],
lw=1)
tpr_fpr_row = []
tpr_fpr_row.append("%s-%s" % (method, target))
for fpr_iter in np.arange(len(x_labels)):
_, min_index = min(
list(zip(abs(fpr - x_labels[fpr_iter]), range(len(fpr)))))
tpr_fpr_row.append('%.2f' % (tpr[min_index] * 100))
tpr_fpr_table.add_row(tpr_fpr_row)
plt.xlim([10 ** -6, 0.1])
plt.ylim([0.30, 1.0])
plt.grid(linestyle='--', linewidth=1)
plt.xticks(x_labels)
plt.yticks(np.linspace(0.30, 1.0, 8, endpoint=True))
plt.xscale('log')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC on IJB')
plt.legend(loc="lower right")
fig.savefig(os.path.join("./", '%s.pdf' % pdf_name))
print(tpr_fpr_table)

```

trying to load your "MS1MV2-ResNet100-ArcFace-TestMode(N1D1F1).npy" and the label from ijbb_template_pair_label.txt to get the test result and draw the ROC curve.

But, I got result like this:
| Methods | 1e-06 | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 |
|:------------------------------------------|---------:|---------:|---------:|---------:|---------:|---------:|
| MS1MV2-ResNet100-ArcFace-TestMode(N1D1F1) | 0.409056 | 0.908082 | 0.947712 | 0.963583 | 0.975463 | 0.986271 |

which is different from [Reformat and speed up IJB evaluation](https://github.com/deepinsight/insightface/pull/1349):
| Methods | 1e-06 | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 |
|:-------------------------------------|---------:|---------:|---------:|---------:|---------:|---------:|
| MS1MV2-ResNet100-Arcface_IJBB_N1D1F1 | 0.408861 | 0.899513 | 0.946349 | 0.964167 | 0.976144 | 0.98666 |

especially when FAR=1e-05,TAR increase almost 2%.
I'm confused, am I missing something?

BTW, when I use [ijb_11.py](https://github.com/deepinsight/insightface/blob/master/recognition/_evaluation_/ijb/ijb_11.py), the above mentioned problem still occurs.

hoping you can help me to figure it out, thank you in advance.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.