facebookresearch / facebookresearch/perception_models
using PE-lang for text-based search
- Dominant language
- Jupyter Notebook
- Stars
- 2.4k
- Forks
- 162
- PR merge metrics
- No merged PRs in 30d
Description
Hello, I am using PE-lang to extract visual encodings using this code snippet:
```
import torch
from PIL import Image
import core.vision_encoder.pe as pe
import core.vision_encoder.transforms as transforms
print("PE configs:", pe.VisionTransformer.available_configs())
# PE configs: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224', 'PE-Core-S16-384', 'PE-Core-T16-384', 'PE-Lang-G14-448', 'PE-Lang-L14-448', 'PE-Lang-G14-448-Tiling', 'PE-Lang-L14-448-Tiling', 'PE-Spatial-G14-448', 'PE-Spatial-L14-448', 'PE-Spatial-B16-512', 'PE-Spatial-S16-512', 'PE-Spatial-T16-512']
model = pe.VisionTransformer.from_config("PE-Lang-L14-448", pretrained=True) # Loads from HF
model = model.cuda()
preprocess = transforms.get_image_transform(model.image_size)
image = preprocess(Image.open("docs/assets/cat.png")).unsqueeze(0).cuda()
out = model.forward_features(image, strip_cls_token=True) # pass layer_idx= to get a specific layer's output!
print(out.shape)
# torch.Size([1, 1024, 1024])
```
and writing the vectors to a vector db.
I need to create a search interface into these saved vectors. How do I fetch the text encodings for say "person walking dog" and query the PE-lang's vectors saved to the vector db.
I am familiar with the approach described here, but how can I go about using PE-Lang which i understand is better suited than PE-Core for VQA and related tasks.
```
import torch
from PIL import Image
import core.vision_encoder.pe as pe
import core.vision_encoder.transforms as transforms
print("CLIP configs:", pe.CLIP.available_configs())
# CLIP configs: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224', 'PE-Core-S16-384', 'PE-Core-T16-384']
model = pe.CLIP.from_config("PE-Core-L14-336", pretrained=True) # Downloads from HF
model = model.cuda()
preprocess = transforms.get_image_transform(model.image_size)
tokenizer = transforms.get_text_tokenizer(model.context_length)
image = preprocess(Image.open("docs/assets/cat.png")).unsqueeze(0).cuda()
text = tokenizer(["a diagram", "a dog", "a cat"]).cuda()
with torch.no_grad(), torch.autocast("cuda"):
image_features, text_features, logit_scale = model(image, text)
text_probs = (logit_scale * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs) # prints: [[0.0, 0.0, 1.0]]
```
Contributor guide
Assessment
This issue has not been assessed yet.