Identifying unique people throughout video

I want to know how many unique people are in this video. https://www.youtube.com/watch?v=XtXJSX71hhg
I know it is 4, but through insight-face, I want to do it and it shows face ids around 10. I analyzed and looked at it, what happens is whenever extremely side-angled faces are detected obviously it will be nonrecognizable with the existing list of faces so it will issue a new id to that, so I tried ignoring all faces with faces.embedding_norm<20 but still the issue persists. so now how can I improve results so it can give max 5-6 id if 4 people are there?