Very high similarity for different face

Hi, I tried using the default and mobilenet build for face recognition but it gave me a very high similarity (0.99++) even though the face is different. I tried using the SubCenter ArcFace build which is more accurate but for the cost of response time (much much slower than default and mobilenet build). Is there any other way to work around this without scarificing the response time?