LAION-AI/CLIP_benchmark

View on GitHub

CLIP-like model evaluation

Stars: 814
Forks: 103
Open beginner issues: 0
Indexed issues: 30

Dominant language: Python
License: MIT
Last GitHub push: Jul 23, 2026
Latest indexed: Sep 18, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

30 open issues indexed

Proposal: optional audit manifest for benchmark result provenance Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

LAION-AI/CLIP_benchmark#153 · Opened May 10, 2026
CTMinfo Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

LAION-AI/CLIP_benchmark#152 · Opened Apr 12, 2026
Zeroshot-classification accuracy function dimensionality bug Open

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

LAION-AI/CLIP_benchmark#147 · Opened Jan 22, 2026
Mismatch between MS-COCO val2014 images and annotations Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

LAION-AI/CLIP_benchmark#145 · Opened Oct 26, 2025
Eval custom open_clip architecture – Add model config Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#144 · Opened Sep 13, 2025
Possibility to save weights after training with linear probing Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#143 · Opened Aug 15, 2025
Implement LP++ for better linear probing results Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#142 · Opened Jul 31, 2025
Webdatasets for Linear Probe evaluation Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

LAION-AI/CLIP_benchmark#127 · 6 comments · Opened May 27, 2024
additional argument not recognized in MODEL_TYPES Open

Difficulty 2/5 1-3 hours Newbie friendliness 58/100

LAION-AI/CLIP_benchmark#123 · Opened Mar 21, 2024
Update babel image net Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

LAION-AI/CLIP_benchmark#117 · Opened Jan 17, 2024
Add support of zero-shot classification/retrieval evaluation of captioning models using log-likelihood scoring Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

LAION-AI/CLIP_benchmark#115 · Opened Jan 4, 2024
Support computing metrics using test time distribution normalisation Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#112 · Opened Nov 1, 2023
Add compositionality benchmarks Open

new datasets

Difficulty 5/5 Over a week Newbie friendliness 28/100

LAION-AI/CLIP_benchmark#97 · 10 comments · Opened Jun 30, 2023
generate diagram as part of the cli Open

important UX

Difficulty 5/5 Over a week Newbie friendliness 20/100

LAION-AI/CLIP_benchmark#80 · 1 comment · Opened Feb 3, 2023
Attribute based eval Open

new datasets

Difficulty 5/5 Over a week Newbie friendliness 20/100

LAION-AI/CLIP_benchmark#72 · 4 comments · 1 reaction · Opened Jan 26, 2023
Evaluate mse aligned multilingual clip Open

new model

Difficulty 3/5 1-2 days Newbie friendliness 35/100

LAION-AI/CLIP_benchmark#69 · 2 comments · Opened Jan 20, 2023
Organize CLI arguments into groups Open

UX

Difficulty 2/5 1-3 hours Newbie friendliness 38/100

LAION-AI/CLIP_benchmark#66 · Opened Jan 13, 2023
Add forensic/exif tasks Open

new datasets

Difficulty 5/5 Over a week Newbie friendliness 20/100

LAION-AI/CLIP_benchmark#65 · Opened Jan 12, 2023
Variation in the benchmark Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 35/100

LAION-AI/CLIP_benchmark#59 · 1 comment · Opened Dec 26, 2022
add multi30k for eval Open

new datasets

Difficulty 4/5 3-5 days Newbie friendliness 35/100

LAION-AI/CLIP_benchmark#46 · 1 comment · Opened Nov 28, 2022
add xm3600 for multilingual eval Open

new datasets

Difficulty 4/5 3-5 days Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#44 · Opened Nov 23, 2022
Support ImageNet-X Open

new datasets

Difficulty 4/5 3-5 days Newbie friendliness 32/100

LAION-AI/CLIP_benchmark#41 · Opened Nov 21, 2022
add multilingual task from mural Open

new datasets

Difficulty 5/5 Over a week Newbie friendliness 20/100

LAION-AI/CLIP_benchmark#35 · Opened Nov 19, 2022
add chinese retrieval tasks Open

new datasets

Difficulty 4/5 3-5 days Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#33 · 1 comment · Opened Nov 18, 2022
Add XMRec multilingual retrieval benchmark Open

new datasets

Difficulty 4/5 3-5 days Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#31 · 1 reaction · Opened Nov 15, 2022
Potential using mis-ordered KITTI prompt Open

new datasets

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

LAION-AI/CLIP_benchmark#21 · 3 comments · 2 reactions · Opened Nov 10, 2022
add fairness related evaluations Open

new datasets

Difficulty 5/5 Over a week Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#15 · 8 comments · Opened Oct 6, 2022
make zero shot classifier building fast by batching Open

UX

Difficulty 3/5 1-2 days Newbie friendliness 42/100

LAION-AI/CLIP_benchmark#5 · 2 comments · Opened Aug 1, 2022
Make a training companion eval script Open

important UX

Difficulty 4/5 3-5 days Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#2 · Opened Jul 31, 2022
support more clip models Open

new model

Difficulty 4/5 3-5 days Newbie friendliness 25/100

LAION-AI/CLIP_benchmark#1 · 18 comments · Opened May 12, 2022

LAION-AI/CLIP_benchmark

Get new issues in your inbox