Owner

open-compass

3 indexed repositories · View on GitHub

opencompass
OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, ...

Python · 7455 stars
VLMEvalKit
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

Python · 4400 stars
AgentCompass
[EMNLP 2026] AgentCompass is an extensible open-source evaluation infrastructure for systematically assessing LLM/VLM agent capabilities.

Python · 135 stars

Beginner friendly Everything

Open All

`gsm8k_postprocess` breaks numeric answers at thousands separators (`\boxed{$9{,}500}` → `500`, `\boxed{8,000}` → `000`) Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

open-compass/opencompass#2647 · 2 comments · Opened Sep 18, 2026
olmOCRBench: `evaluate()` returns `None`, so the score never reaches `status.json` Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

open-compass/VLMEvalKit#1690 · 3 comments · Opened Sep 16, 2026
`dump()` silently drops predictions that begin with a URL and exceed 2079 characters Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

open-compass/VLMEvalKit#1689 · Opened Sep 16, 2026
DOCLING / IDEFICS2 / Mantis / XGenMM fail to import on transformers 5.x: AutoModelForVision2Seq was removed Open

Difficulty 2/5 1-3 hours Newbie friendliness 85/100

open-compass/VLMEvalKit#1675 · Opened Sep 8, 2026
TopkRetriever crashes on transformers 5.x: tokenizer.encode_plus was removed Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

open-compass/opencompass#2635 · Opened Sep 8, 2026
Grading integrity: model code rewrites ground truth in three benchmarks; CRUXEval splices prediction into assertion; unparseable verdicts exit denominator; LCEvaluator eval() on raw prediction (follow-up to #2535) Open

open-compass/opencompass#2631 · 1 assignee · Opened Sep 3, 2026
[Bug] ToxicEvaluator: expected_max_toxicity uses built-in max() on a NaN-containing array, so the metric depends on sample order Open

open-compass/opencompass#2627 · 1 assignee · Opened Sep 1, 2026
Structured extra_records output is discarded for the entire dataset if any single sample fails Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

open-compass/VLMEvalKit#1665 · 1 comment · Opened Sep 1, 2026
[Bug] 内置 ACTF 消费者无法一致解析非 OpenAI 工具调用 / Built-in ACTF consumers silently ignore non-OpenAI tool-call payloads Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

open-compass/AgentCompass#288 · 2 comments · Opened Aug 31, 2026
OpenAI-compat: GPT4V api_base is the full /v1/chat/completions URL, not /v1 Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

open-compass/VLMEvalKit#1661 · 2 comments · Opened Aug 28, 2026
OpenAI-compat: OpenAISDK openai_api_base /v1, not the requests OpenAI full-path default Open

open-compass/opencompass#2615 · 1 assignee · Opened Aug 28, 2026
Interop idea: a VLMEvalKit-openeval-adapter for portable eval results Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1652 · Opened Aug 23, 2026
Feature request: EvalPort import/export for CustomDataset (adapter already built and tested) Open

open-compass/opencompass#2604 · 1 assignee · Opened Aug 21, 2026
[Bug] reconstruct_run_result drops StepInfo.tools from persisted trajectories Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

open-compass/AgentCompass#276 · Opened Aug 20, 2026
[Bug] pass@k and avg@k collapse to attempt-1 accuracy in micro-weighted aggregation Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

open-compass/AgentCompass#273 · 1 comment · Opened Aug 20, 2026
Add Pluto-AI-Labs/Apollo-VL-Edge-3B to Leaderboard Open

Difficulty 3/5 1-2 days Newbie friendliness 64/100

open-compass/VLMEvalKit#1632 · Opened Aug 14, 2026
Qwen2.5-VL-7B模型，对评测集LogicVista，从推理结果中提取答案错误，导致误判 Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1621 · Opened Aug 6, 2026
MMMU Open-ended问题无法正确评测 Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1617 · Opened Aug 4, 2026
OpenVLM: one-decimal Overall makes the board un-checkable, and hides that 8 boards' #1 leads are under 1.2 sigma Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

open-compass/VLMEvalKit#1615 · 1 comment · Opened Aug 1, 2026
CompassAcademic: 4 of 7 objective boards have an unseparated #1, two are exact ties broken by display order Open

open-compass/opencompass#2574 · 1 comment · 1 assignee · Opened Aug 1, 2026
[Bug] Qwen3.5 cannot run when using HuggingFacewithChatTemplate Open

open-compass/opencompass#2573 · 1 assignee · Opened Jul 31, 2026
有些基准没有放数据url Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

open-compass/VLMEvalKit#1614 · Opened Jul 31, 2026
LiveMMBench空链接 Open

Difficulty 3/5 1-2 days Newbie friendliness 38/100

open-compass/VLMEvalKit#1600 · Opened Jul 16, 2026
MedFailBench integration proposal for clinical AI safety evaluation Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

open-compass/opencompass#2516 · 2 comments · Opened Jul 8, 2026
请问flyai-vl模型的评测什么时候可以支持？ Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1598 · Opened Jul 7, 2026
[Feature] DIK-structured Turkish syntactic dependency annotation dataset for structured linguistic evaluation Open

open-compass/opencompass#2508 · 1 comment · 1 assignee · Opened Jul 3, 2026
[Bug] run.py: LOCAL_RANK defaults to 1 instead of 0 Open

Difficulty 1/5 Under an hour Newbie friendliness 78/100

open-compass/VLMEvalKit#1589 · 1 comment · Opened Jun 28, 2026
[Bug] Thyme: sandbox image fed in assistant turn + KV-cache reused across re-encoded turns corrupts multi-round vision context Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

open-compass/VLMEvalKit#1590 · 1 comment · Opened Jun 28, 2026
Adding [BenchCAD]: would you accept execution-based image→CadQuery-code tasks (heavier deps)? Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

open-compass/VLMEvalKit#1582 · Opened Jun 25, 2026
[Bug] CharXiv_descriptive_val DATASET_MD5 appears to be incorrect after #1555 Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

open-compass/VLMEvalKit#1580 · Opened Jun 23, 2026
[Feature] 集成 ThinkCheck 作为 OpenCompass 的“推理健康度”评估维度 Open

open-compass/opencompass#2475 · 1 assignee · Opened Jun 8, 2026
Qwen 3.5支持 Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

open-compass/VLMEvalKit#1562 · 3 reactions · Opened Jun 1, 2026
SEEDBench_IMG & MMStar: option text "None" parsed as NaN by pandas Open

Difficulty 3/5 1-2 days Newbie friendliness 64/100

open-compass/VLMEvalKit#1559 · Opened May 28, 2026
Ovis2.6-80B-A3B 评测模型预计什么时候可以支持？ Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1553 · Opened May 25, 2026
是否能发布pypi源 Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

open-compass/VLMEvalKit#1537 · Opened May 8, 2026
model：llava-1.5-7b，data：mmebench Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1529 · Opened Apr 27, 2026
Confirm whether Polygon3 is still required Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

open-compass/VLMEvalKit#1528 · Opened Apr 27, 2026
About MMBench Evaluation Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

open-compass/VLMEvalKit#1516 · 3 comments · 3 reactions · Opened Apr 16, 2026
[Feature] TruthfulQA mcq模式评测支持 Open

open-compass/opencompass#2427 · 1 comment · 1 assignee · Opened Mar 31, 2026
[Bug] [Feature] 评测框架代码过于耦合，失误得重新跑推理，文档中没有看到可以复用推理结果从新评测的命令 Open

open-compass/opencompass#2424 · 1 comment · 1 assignee · Opened Mar 30, 2026
How to evaluate my fine-tuned Qwen-VL model on a locally downloaded dataset? Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

open-compass/VLMEvalKit#1500 · Opened Mar 28, 2026
The tsv file is in {data_root}, but the md5 does not match, will re-download in MathCanvas Open

Difficulty 3/5 1-2 days Newbie friendliness 32/100

open-compass/VLMEvalKit#1498 · Opened Mar 25, 2026
3DSRBench和SiteBenchImage两个数据集评测的bug麻烦修复下，修复方式文中给出。 Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

open-compass/VLMEvalKit#1485 · Opened Mar 17, 2026
[Feature] opencompass公开学术多模态榜单公布qwen3.5模型的结果 Open

open-compass/opencompass#2413 · 1 assignee · Opened Mar 13, 2026
评测MMBench数据集发现部分问题判别错误 Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

open-compass/VLMEvalKit#1480 · 2 comments · Opened Mar 11, 2026
是否支持数据集指定索引的评测？ Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1474 · Opened Mar 9, 2026
是否支持语音模型的评测 Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

open-compass/VLMEvalKit#1462 · Opened Mar 3, 2026
【OmniDocBench】Recommend updating the benchmark version Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

open-compass/VLMEvalKit#1456 · 5 comments · Opened Feb 26, 2026
[Feature] Question: where would a long horizon "tension crash test" TXT pack fit in OpenCompass? Open

open-compass/opencompass#2396 · 1 comment · 1 assignee · Opened Feb 6, 2026
Suggest to integrate the BabyVision dataset Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

open-compass/VLMEvalKit#1438 · 2 comments · 3 reactions · Opened Feb 6, 2026

Showing the newest 100

This page lists what was indexed most recently. Advanced filter has the whole inventory, narrowed by language, difficulty and how long a task takes.

Open advanced filter

open-compass

Showing the newest 100

Get new issues in your inbox