open-compass/VLMEvalKit

View on GitHub

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Stars: 4.4k
Forks: 768
Open beginner issues: 6
Indexed issues: 224
Avg merge: 1d 10h
Merged PRs (30d): 17

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 18, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

224 open issues indexed

olmOCRBench: `evaluate()` returns `None`, so the score never reaches `status.json` Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

open-compass/VLMEvalKit#1690 · 3 comments · Opened Sep 16, 2026
`dump()` silently drops predictions that begin with a URL and exceed 2079 characters Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

open-compass/VLMEvalKit#1689 · Opened Sep 16, 2026
DOCLING / IDEFICS2 / Mantis / XGenMM fail to import on transformers 5.x: AutoModelForVision2Seq was removed Open

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

open-compass/VLMEvalKit#1675 · Opened Sep 8, 2026
Structured extra_records output is discarded for the entire dataset if any single sample fails Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

open-compass/VLMEvalKit#1665 · 1 comment · Opened Sep 1, 2026
OpenAI-compat: GPT4V api_base is the full /v1/chat/completions URL, not /v1 Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

open-compass/VLMEvalKit#1661 · 2 comments · Opened Aug 28, 2026
Interop idea: a VLMEvalKit-openeval-adapter for portable eval results Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1652 · Opened Aug 23, 2026
Add Pluto-AI-Labs/Apollo-VL-Edge-3B to Leaderboard Open

Difficulty 3/5 1-2 days Newbie friendliness 64/100

open-compass/VLMEvalKit#1632 · Opened Aug 14, 2026
Qwen2.5-VL-7B模型，对评测集LogicVista，从推理结果中提取答案错误，导致误判 Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1621 · Opened Aug 6, 2026
MMMU Open-ended问题无法正确评测 Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1617 · Opened Aug 4, 2026
OpenVLM: one-decimal Overall makes the board un-checkable, and hides that 8 boards' #1 leads are under 1.2 sigma Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

open-compass/VLMEvalKit#1615 · 1 comment · Opened Aug 1, 2026
有些基准没有放数据url Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1614 · Opened Jul 31, 2026
LiveMMBench空链接 Open

Difficulty 3/5 1-2 days Newbie friendliness 38/100

open-compass/VLMEvalKit#1600 · Opened Jul 16, 2026
请问flyai-vl模型的评测什么时候可以支持？ Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1598 · Opened Jul 7, 2026
[Bug] run.py: LOCAL_RANK defaults to 1 instead of 0 Open

Difficulty 1/5 Under an hour Newbie friendliness 78/100

open-compass/VLMEvalKit#1589 · 1 comment · Opened Jun 28, 2026
[Bug] Thyme: sandbox image fed in assistant turn + KV-cache reused across re-encoded turns corrupts multi-round vision context Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

open-compass/VLMEvalKit#1590 · 1 comment · Opened Jun 28, 2026
Adding [BenchCAD]: would you accept execution-based image→CadQuery-code tasks (heavier deps)? Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

open-compass/VLMEvalKit#1582 · Opened Jun 25, 2026
[Bug] CharXiv_descriptive_val DATASET_MD5 appears to be incorrect after #1555 Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

open-compass/VLMEvalKit#1580 · Opened Jun 23, 2026
Qwen 3.5支持 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

open-compass/VLMEvalKit#1562 · 3 reactions · Opened Jun 1, 2026
SEEDBench_IMG & MMStar: option text "None" parsed as NaN by pandas Open

Difficulty 3/5 1-2 days Newbie friendliness 64/100

open-compass/VLMEvalKit#1559 · Opened May 28, 2026
Ovis2.6-80B-A3B 评测模型预计什么时候可以支持？ Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1553 · Opened May 25, 2026
是否能发布pypi源 Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

open-compass/VLMEvalKit#1537 · Opened May 8, 2026
model：llava-1.5-7b，data：mmebench Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1529 · Opened Apr 27, 2026
Confirm whether Polygon3 is still required Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

open-compass/VLMEvalKit#1528 · Opened Apr 27, 2026
About MMBench Evaluation Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

open-compass/VLMEvalKit#1516 · 3 comments · 3 reactions · Opened Apr 16, 2026
How to evaluate my fine-tuned Qwen-VL model on a locally downloaded dataset? Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

open-compass/VLMEvalKit#1500 · Opened Mar 28, 2026
The tsv file is in {data_root}, but the md5 does not match, will re-download in MathCanvas Open

Difficulty 3/5 1-2 days Newbie friendliness 32/100

open-compass/VLMEvalKit#1498 · Opened Mar 25, 2026
3DSRBench和SiteBenchImage两个数据集评测的bug麻烦修复下，修复方式文中给出。 Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

open-compass/VLMEvalKit#1485 · Opened Mar 17, 2026
评测MMBench数据集发现部分问题判别错误 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1480 · 2 comments · Opened Mar 11, 2026
是否支持数据集指定索引的评测？ Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1474 · Opened Mar 9, 2026
是否支持语音模型的评测 Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

open-compass/VLMEvalKit#1462 · Opened Mar 3, 2026
【OmniDocBench】Recommend updating the benchmark version Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

open-compass/VLMEvalKit#1456 · 5 comments · Opened Feb 26, 2026
Suggest to integrate the BabyVision dataset Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1438 · 2 comments · 3 reactions · Opened Feb 6, 2026
自定义数据集功能，能够支持音频和视频 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

open-compass/VLMEvalKit#1417 · Opened Jan 27, 2026
Qwen2.5-VL-3B model evaluating BLINK, and it reported an OOM error: Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1416 · Opened Jan 25, 2026
evaluate Video-MME_1fps on SmolVLM got cuda OOM error Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1401 · Opened Jan 12, 2026
[MMSIBench] OSError: image file is truncated Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1400 · Opened Jan 10, 2026
screenspot 分数对齐 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1391 · 1 comment · Opened Dec 30, 2025
Video Holmes llm judge is not used in evaluation? Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

open-compass/VLMEvalKit#1390 · Opened Dec 30, 2025
Qwen3-VL-4B-Instruct模型评估ocrbench结果和官方公布结果不一致 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1389 · 3 comments · Opened Dec 30, 2025
模型评估结果和官方公布结果不一致 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1388 · Opened Dec 29, 2025
我的环境现在只支持我要使用的模型，但VLMEvalKit 在导入时尝试加载所有模型的依赖 Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1383 · Opened Dec 27, 2025
如何使用提前下载好的本地数据集，在哪里能配置路径呢 Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1373 · 1 comment · Opened Dec 24, 2025
Update evaluation records of InternVL3 Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1372 · Opened Dec 24, 2025
Reproducibility of deepseek-vl inference results Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1367 · 4 comments · Opened Dec 19, 2025
Hope to include Qwen3 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

open-compass/VLMEvalKit#1366 · 1 comment · Opened Dec 18, 2025
How can I use VLMEvalKit to test the DeepSeek-OCR model and obtain its accuracy score on OmniDocBench? Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

open-compass/VLMEvalKit#1361 · 1 comment · Opened Dec 16, 2025
Hope to open slack for everyone to communicate Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

open-compass/VLMEvalKit#1337 · Opened Dec 3, 2025
MMVet 数据集问题 Open

Difficulty 1/5 Under an hour Newbie friendliness 28/100

open-compass/VLMEvalKit#1336 · 1 comment · Opened Dec 2, 2025
Qwen3-VL on ScreenSpot datasets Open

Difficulty 3/5 1-2 days Newbie friendliness 42/100

open-compass/VLMEvalKit#1332 · 2 comments · Opened Nov 28, 2025
Unable to reproduce Intern-S1-mini on XLRS-Bench-lite Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

open-compass/VLMEvalKit#1331 · Opened Nov 27, 2025

open-compass/VLMEvalKit

Get new issues in your inbox