facebookresearch / facebookresearch/perception_models

using PE-lang for text-based search

Open
#99 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.4k
Forks
162
PR merge metrics
No merged PRs in 30d

Description

Hello, I am using PE-lang to extract visual encodings using this code snippet:

```
import torch
from PIL import Image
import core.vision_encoder.pe as pe
import core.vision_encoder.transforms as transforms

print("PE configs:", pe.VisionTransformer.available_configs())
# PE configs: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224', 'PE-Core-S16-384', 'PE-Core-T16-384', 'PE-Lang-G14-448', 'PE-Lang-L14-448', 'PE-Lang-G14-448-Tiling', 'PE-Lang-L14-448-Tiling', 'PE-Spatial-G14-448', 'PE-Spatial-L14-448', 'PE-Spatial-B16-512', 'PE-Spatial-S16-512', 'PE-Spatial-T16-512']

model = pe.VisionTransformer.from_config("PE-Lang-L14-448", pretrained=True) # Loads from HF
model = model.cuda()

preprocess = transforms.get_image_transform(model.image_size)
image = preprocess(Image.open("docs/assets/cat.png")).unsqueeze(0).cuda()

out = model.forward_features(image, strip_cls_token=True) # pass layer_idx= to get a specific layer's output!
print(out.shape)
# torch.Size([1, 1024, 1024])
```
and writing the vectors to a vector db.

I need to create a search interface into these saved vectors. How do I fetch the text encodings for say "person walking dog" and query the PE-lang's vectors saved to the vector db.

I am familiar with the approach described here, but how can I go about using PE-Lang which i understand is better suited than PE-Core for VQA and related tasks.
```
import torch
from PIL import Image
import core.vision_encoder.pe as pe
import core.vision_encoder.transforms as transforms

print("CLIP configs:", pe.CLIP.available_configs())
# CLIP configs: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224', 'PE-Core-S16-384', 'PE-Core-T16-384']

model = pe.CLIP.from_config("PE-Core-L14-336", pretrained=True) # Downloads from HF
model = model.cuda()

preprocess = transforms.get_image_transform(model.image_size)
tokenizer = transforms.get_text_tokenizer(model.context_length)

image = preprocess(Image.open("docs/assets/cat.png")).unsqueeze(0).cuda()
text = tokenizer(["a diagram", "a dog", "a cat"]).cuda()

with torch.no_grad(), torch.autocast("cuda"):
image_features, text_features, logit_scale = model(image, text)
text_probs = (logit_scale * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs) # prints: [[0.0, 0.0, 1.0]]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.