deepinsight / deepinsight/insightface

Using vggface2 based embedding in inswapper is not prducing correct results

Open
#2,655 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29.7k
Forks
6.1k
PR merge metrics
No merged PRs in 30d

Description

Hi,

I'm trying to use vggface2 based embedding using [facenet-pytorch](https://github.com/timesler/facenet-pytorch)
Here is the code to generate:
```python

import torch
from facenet_pytorch import MTCNN, InceptionResnetV1

class FaceEmbed:
'''https://github.com/timesler/facenet-pytorch'''
def __init__(self):

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# If required, create a face detection pipeline using MTCNN:
self.mtcnn = MTCNN(device=device)

# Create an inception resnet (in eval mode):
self.resnet = InceptionResnetV1(pretrained='vggface2').eval()

def get(self, img):
# Get cropped and prewhitened image tensor
img_cropped = self.mtcnn(img)

# Calculate embedding (unsqueeze to add batch dimension)
img_embedding = self.resnet(img_cropped.unsqueeze(0))

return img_embedding.detach().numpy().flatten()
```
Then using a custom inswapper, I have replaced this line
https://github.com/deepinsight/insightface/blob/da738727d5fe26d305499c311bd05cb51c3936d5/python-package/insightface/model_zoo/inswapper.py#L50
to
```
latent = embed.get(source_image).reshape((1,-1))
```
![result](https://github.com/user-attachments/assets/ad976086-4603-4307-ac5f-4288ebb4dc82)

The resulting swap (Incorrect in picture) is not at all good. Any pointers?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with python-package/insightface/model_zoo/inswapper.py at the embedding assignment around line 50, then compare the facenet-pytorch VGGFace2 preprocessing and output with the embedding format expected by inswapper. Reproduce the reported swap using the supplied FaceEmbed code and image; done means the custom embedding produces a correct result or the incompatibility is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.