open-compass/VLMEvalKit

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Stars: 4.4k
Forks: 768
Open beginner issues: 6
Indexed issues: 224
Avg merge: 1d 10h
Merged PRs (30d): 17

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 20, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

6 beginner-friendly issues open

olmOCRBench: `evaluate()` returns `None`, so the score never reaches `status.json` Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

open-compass/VLMEvalKit#1690 · 3 comments · Opened Sep 16, 2026
`dump()` silently drops predictions that begin with a URL and exceed 2079 characters Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

open-compass/VLMEvalKit#1689 · Opened Sep 16, 2026
DOCLING / IDEFICS2 / Mantis / XGenMM fail to import on transformers 5.x: AutoModelForVision2Seq was removed Open

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

open-compass/VLMEvalKit#1675 · Opened Sep 8, 2026
[Bug] run.py: LOCAL_RANK defaults to 1 instead of 0 Open

Difficulty 1/5 Under an hour Newbie friendliness 78/100

open-compass/VLMEvalKit#1589 · 1 comment · Opened Jun 28, 2026
[Bug] CharXiv_descriptive_val DATASET_MD5 appears to be incorrect after #1555 Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

open-compass/VLMEvalKit#1580 · Opened Jun 23, 2026
Confirm whether Polygon3 is still required Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

open-compass/VLMEvalKit#1528 · Opened Apr 27, 2026

A short digest of beginner-friendly GitHub issues.