deepinsight / deepinsight/insightface
Why the evaluation result on IJB-B is different from paper&your test(almost same code)?
- Dominant language
- Python
- Stars
- 29.7k
- Forks
- 6.1k
- PR merge metrics
- No merged PRs in 30d
Description
First of all, thank you for your excellent works, I learn a lot from your repository.
here is my puzzle:
my code is :
```
import os
import matplotlib
matplotlib.use('Agg')
from pathlib import Path
import pandas as pd
import numpy as np
from prettytable import PrettyTable
import matplotlib.pyplot as plt
from sklearn.metrics import roc_curve, auc
from menpo.visualize.viewmatplotlib import sample_colours_from_colourmap
def read_template_pair_list(path):
# pairs = np.loadtxt(path, dtype=str)
pairs = pd.read_csv(path, sep=' ', header=None).values
# print(pairs.shape)
# print(pairs[:, 0].astype(np.int))
t1 = pairs[:, 0].astype(np.int)
t2 = pairs[:, 1].astype(np.int)
label = pairs[:, 2].astype(np.int)
return t1, t2, label
if __name__ == "__main__":
score_save_file = r"I:\Dataset\IJB_release\IJBB\result\MS1MV2-ResNet100-ArcFace-TestMode(N1D1F1).npy"
#MS1MV2-ResNet100-Arcface_IJBB_N1D1F1
image_path = "./IJBB"
pdf_name = "MS1MV2-ResNet100-ArcFace"
target = "IJBB"
p1, p2, label = read_template_pair_list(
os.path.join(r"I:\Dataset\IJB_release\IJBB\meta", #'%s/meta' % image_path,
'%s_template_pair_label.txt' % target.lower()))
files = [score_save_file]
methods = []
scores = []
for file in files:
methods.append(Path(file).stem)
scores.append(np.load(file))
methods = np.array(methods)
scores = dict(zip(methods, scores))
colours = dict(
zip(methods, sample_colours_from_colourmap(methods.shape[0], 'Set2')))
x_labels = [10 ** -6, 10 ** -5, 10 ** -4, 10 ** -3, 10 ** -2, 10 ** -1]
#x_labels = [10 ** -5, 10 ** -4, 10 ** -3, 10 ** -2, 10 ** -1]
tpr_fpr_table = PrettyTable(['Methods'] + [str(x) for x in x_labels])
fig = plt.figure()
for method in methods:
fpr, tpr, _ = roc_curve(label, scores[method])
roc_auc = auc(fpr, tpr)
fpr = np.flipud(fpr)
tpr = np.flipud(tpr) # select largest tpr at same fpr
plt.plot(fpr,
tpr,
color=colours[method],
lw=1)
tpr_fpr_row = []
tpr_fpr_row.append("%s-%s" % (method, target))
for fpr_iter in np.arange(len(x_labels)):
_, min_index = min(
list(zip(abs(fpr - x_labels[fpr_iter]), range(len(fpr)))))
tpr_fpr_row.append('%.2f' % (tpr[min_index] * 100))
tpr_fpr_table.add_row(tpr_fpr_row)
plt.xlim([10 ** -6, 0.1])
plt.ylim([0.30, 1.0])
plt.grid(linestyle='--', linewidth=1)
plt.xticks(x_labels)
plt.yticks(np.linspace(0.30, 1.0, 8, endpoint=True))
plt.xscale('log')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC on IJB')
plt.legend(loc="lower right")
fig.savefig(os.path.join("./", '%s.pdf' % pdf_name))
print(tpr_fpr_table)
```
trying to load your "MS1MV2-ResNet100-ArcFace-TestMode(N1D1F1).npy" and the label from ijbb_template_pair_label.txt to get the test result and draw the ROC curve.
But, I got result like this:
| Methods | 1e-06 | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 |
|:------------------------------------------|---------:|---------:|---------:|---------:|---------:|---------:|
| MS1MV2-ResNet100-ArcFace-TestMode(N1D1F1) | 0.409056 | 0.908082 | 0.947712 | 0.963583 | 0.975463 | 0.986271 |
which is different from [Reformat and speed up IJB evaluation](https://github.com/deepinsight/insightface/pull/1349):
| Methods | 1e-06 | 1e-05 | 0.0001 | 0.001 | 0.01 | 0.1 |
|:-------------------------------------|---------:|---------:|---------:|---------:|---------:|---------:|
| MS1MV2-ResNet100-Arcface_IJBB_N1D1F1 | 0.408861 | 0.899513 | 0.946349 | 0.964167 | 0.976144 | 0.98666 |
especially when FAR=1e-05,TAR increase almost 2%.
I'm confused, am I missing something?
BTW, when I use [ijb_11.py](https://github.com/deepinsight/insightface/blob/master/recognition/_evaluation_/ijb/ijb_11.py), the above mentioned problem still occurs.
hoping you can help me to figure it out, thank you in advance.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.