Plugin explore image caption models

Image Summary Plugin uses LLM mutimodal feature to describe images. however this needs GPU

can we use a pipeline approach to first scan images get a caption , for example find all images with a face or a white car and then this subset can feed into above Image summary if more details is desired.

so goal here is get a caption from an image using less GPU intensive approach.

Use CLIP for the initial, flexible, zero-shot searching and filtering of image databases. This helps investigators quickly narrow down a massive amount of data.

Use a lightweight version of LLaVA for conversational, higher-level analysis, such as generating descriptions or answering questions about specific images that were flagged by CLIP.

bliva

lavis
and
https://github.com/salesforce/BLIP

refer rescuebox wiki and hello-world-plugin

DETR

vision transformers