Grouping a collection of faces by person

I have a collection of face encodings, and I'd like to group them together (so, ideally, all the images of person 1 are in one group, all the images of person 2 are in a second group, etc). I'm working with the Python API.

My code currently works, to do this, but it's very slow (which will not surprise you when you see how it works). The algorithm is, basically:

for enc in big_list_of_encodings:
  for group in existing_groups:
    distances = fe.face_distance(group, enc)
    if more than half of distances < 0.6:
      # this matches most of this group: add it
      group.append(enc)
  if we matched no groups:
    # add a new group with just us in it
    existing_groups.append([enc])

This works very well indeed, which was a pleasant shock! However, it's pretty darn slow, not surprisingly; it walks over all the images again and again and again. What I don't know is: is there a better way to do this? I did look at the actual code, but I don't really understand about comparing large numpy vectors and so on, so I thought maybe there's a simpler way that I just don't know about to group together a big collection of vectors based on distance.

Start with the Python API and the face_distance calls described in the issue; review how the collection of encodings is currently compared. A useful resolution would identify a faster supported grouping approach and document its usage and performance, but the issue names no files, entry points, or tests.