deepinsight / deepinsight/insightface
Using vggface2 based embedding in inswapper is not prducing correct results
- Dominant language
- Python
- Stars
- 29.7k
- Forks
- 6.1k
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I'm trying to use vggface2 based embedding using [facenet-pytorch](https://github.com/timesler/facenet-pytorch)
Here is the code to generate:
```python
import torch
from facenet_pytorch import MTCNN, InceptionResnetV1
class FaceEmbed:
'''https://github.com/timesler/facenet-pytorch'''
def __init__(self):
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
# If required, create a face detection pipeline using MTCNN:
self.mtcnn = MTCNN(device=device)
# Create an inception resnet (in eval mode):
self.resnet = InceptionResnetV1(pretrained='vggface2').eval()
def get(self, img):
# Get cropped and prewhitened image tensor
img_cropped = self.mtcnn(img)
# Calculate embedding (unsqueeze to add batch dimension)
img_embedding = self.resnet(img_cropped.unsqueeze(0))
return img_embedding.detach().numpy().flatten()
```
Then using a custom inswapper, I have replaced this line
https://github.com/deepinsight/insightface/blob/da738727d5fe26d305499c311bd05cb51c3936d5/python-package/insightface/model_zoo/inswapper.py#L50
to
```
latent = embed.get(source_image).reshape((1,-1))
```

The resulting swap (Incorrect in picture) is not at all good. Any pointers?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with python-package/insightface/model_zoo/inswapper.py at the embedding assignment around line 50, then compare the facenet-pytorch VGGFace2 preprocessing and output with the embedding format expected by inswapper. Reproduce the reported swap using the supplied FaceEmbed code and image; done means the custom embedding produces a correct result or the incompatibility is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100