model architectures and pretrained models to support

## installable

- [ ] https://github.com/salesforce/LAVIS
- https://github.com/salesforce/BLIP
- https://github.com/salesforce/ALBEF
- [ ] https://github.com/facebookresearch/multimodal
- FLAVA
- LateFusion
- ALBEF
- MDETR
- OMNIVORE
- video-gpt
- [ ] https://github.com/ai-forever/ru-clip
- [ ] https://github.com/FreddeFrallan/Multilingual-CLIP
- [ ] https://github.com/lucidrains/x-clip
- see citations for supported pre-trained checkpoints
- [ ] https://github.com/ai-forever/ru-dolph
- [ ] https://github.com/optas/artemis
- [ ] https://github.com/CompVis/net2net
- [ ] LiT - https://github.com/google-research/vision_transformer/
- [ ] (old) - DAMSM - https://github.com/taoxugit/AttnGAN
- [ ] https://github.com/facebookresearch/vilbert-multi-task
- [ ] https://github.com/Aleph-Alpha/magma
- [ ] https://github.com/sallymmx/ActionCLIP
- [ ] https://github.com/facebookresearch/mmf
- [ ] https://github.com/rinnakk/japanese-clip
- [ ] https://github.com/j-min/clip-caption-reward
- [ ] https://github.com/facebookresearch/omnivore
- [ ] https://github.com/microsoft/GLIP
- [ ] https://github.com/microsoft/RegionCLIP
- setup.py is actually for detectron2
- [ ] https://github.com/yusufani/TrCLIP
- [ ] https://github.com/facebookresearch/detectron2

## installable with extra effort
- [ ] https://github.com/Sense-GVT/DeCLIP
- installs into a generic "prototype" package that could conflict with other packages using the same "spring" starter template
- dataclasses==0.8 in requirements.txt throws version conflict

## Not installable

- [ ] https://github.com/Kaushalya/medclip
- [ ] https://github.com/clip-italian/clip-italian
- [ ] https://github.com/j-min/VL-T5
- [ ] https://github.com/ashkamath/mdetr
- [ ] https://github.com/drboog/Lafite
- [ ] https://github.com/pbaylies/Augmented_CLIP
- [ ] (old) - https://github.com/sidward14/Style-AttnGAN
- [ ] https://github.com/luogen1996/MCN
- [ ] https://github.com/luogen1996/SimREC
- [ ] https://github.com/facebookresearch/Detic
- [ ] https://github.com/henghuiding/Vision-Language-Transformer
- [ ] https://github.com/allenai/reclip
- [ ] https://github.com/ChenRocks/UNITER
- [ ] https://github.com/raoyongming/DenseCLIP
- [ ] https://github.com/ttlmh/Bridge-Prompt
- [ ] https://github.com/sonoisa/clip-japanese
- [ ] https://github.com/facebookresearch/OTTER
- [ ] https://socraticmodels.github.io/
- [ ] https://github.com/yxuansu/MAGIC
- [ ] https://github.com/mindspore-ai/models/tree/master/research/mm/wukong | https://wukong-dataset.github.io/wukong-dataset/benchmark.html
- [ ] https://github.com/microsoft/UniCL
- [ ] https://github.com/goel-shashank/CyCLIP
- [ ] https://github.com/FacePerceiver/FaRL#use-farl-as-faceclip
- [ ] https://github.com/Lednik7/CLIP-ONNX/tree/main/clip_onnx
- [ ] https://github.com/OFA-Sys/OFA

## Not released

- [ ] ALIGN - https://arxiv.org/abs/2102.05918
- [ ] https://github.com/kakaobrain/coyo-align
- [ ] NUWA
- [ ] https://github.com/microsoft/NUWA
- [ ] https://github.com/lucidrains/nuwa-pytorch
- [ ] CoCa - https://arxiv.org/abs/2205.01917

----

- [x] CLIP
- [x] [OpenAI/CLIP](https://github.com/openai/CLIP)
- [x] [mlfoundations/open_clip](https://github.com/mlfoundations/open_clip)
- [x] CLOOB
- [x] [crowsonkb/cloob-training](https://github.com/crowsonkb/cloob-training)
- [ ] pretrained model from CLOOB authors: https://github.com/ml-jku/cloob/blob/master/src/notebooks/zeroshot.ipynb
- [x] SLIP
- [x] [facebookresearch/SLIP](https://github.com/facebookresearch/SLIP)
- [ ] CLIP-JAX
- [ ] AudioCLIP
- https://github.com/AndreyGuzhov/AudioCLIP
- https://github.com/NotNANtoN/AudioCLIP

- [x] CLIPfa (farsi) - https://github.com/sajjjadayobi/CLIPfa
- [ ] CLIP pretrained on FOOD101 by PASSL? - https://github.com/PaddlePaddle/PASSL/blob/main/docs/Train_CLIP_model.md

- [x] SBERT Multilingual CLIP - https://www.sbert.net/docs/pretrained_models.html#image-text-models

References for more variants:

https://paperswithcode.com/paper/learning-transferable-visual-models-from

Potentially in scope, lower priority
* https://github.com/j-min/VL-T5
* https://github.com/ashkamath/mdetr
* https://github.com/sberbank-ai/ru-dolph
* https://github.com/drboog/Lafite
* https://github.com/pbaylies/Augmented_CLIP
* ALIGN (is this even public? or just big?)
- https://ai.googleblog.com/2021/05/align-scaling-up-visual-and-vision.html
- https://arxiv.org/abs/2102.05918
* ALBEF
- https://arxiv.org/abs/2102.05918
- https://github.com/salesforce/ALBEF
- https://github.com/salesforce/ALBEF/blob/main/visualization.ipynb
* ArtEmis - Affective language for Visual Art
- https://github.com/optas/artemis
* https://github.com/CompVis/net2net
* LiT (not sure if there are public pre-trained models) - https://arxiv.org/abs/2111.07991
* https://colab.research.google.com/github/google-research/vision_transformer/blob/main/lit.ipynb
* NUWA (very unlikely they'll release a reptrained model)
- https://github.com/microsoft/NUWA
- https://github.com/lucidrains/nuwa-pytorch

Older stuff
* DAMSM - https://github.com/taoxugit/AttnGAN
* https://github.com/sidward14/Style-AttnGAN
* https://github.com/luogen1996/MCN
* ViLBERT - https://github.com/facebookresearch/vilbert-multi-task
* VL-BERT - https://github.com/jackroos/VL-BERT

VQA is sort of a generalization of vision language co-training... TBD.
* https://github.com/facebookresearch/Detic
* https://github.com/henghuiding/Vision-Language-Transformer

MAGMA could be another useful approach to promote multi-lingual support

https://github.com/Aleph-Alpha/magma