open-compass/opencompass

View on GitHub

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

Stars: 7.5k
Forks: 869
Open beginner issues: 2
Indexed issues: 290
Avg merge: 17h 52m
Merged PRs (30d): 13

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 16, 2026
Latest indexed: Sep 19, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: good first issue

Beginner friendly Everything

Open All

290 open issues indexed

`gsm8k_postprocess` breaks numeric answers at thousands separators (`\boxed{$9{,}500}` → `500`, `\boxed{8,000}` → `000`) Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

open-compass/opencompass#2647 · 2 comments · Opened Sep 18, 2026
TopkRetriever crashes on transformers 5.x: tokenizer.encode_plus was removed Open

Difficulty 2/5 1-3 hours Newbie friendliness 75/100

open-compass/opencompass#2635 · Opened Sep 8, 2026
Grading integrity: model code rewrites ground truth in three benchmarks; CRUXEval splices prediction into assertion; unparseable verdicts exit denominator; LCEvaluator eval() on raw prediction (follow-up to #2535) Open

open-compass/opencompass#2631 · 1 assignee · Opened Sep 3, 2026
[Bug] ToxicEvaluator: expected_max_toxicity uses built-in max() on a NaN-containing array, so the metric depends on sample order Open

open-compass/opencompass#2627 · 1 assignee · Opened Sep 1, 2026
OpenAI-compat: OpenAISDK openai_api_base /v1, not the requests OpenAI full-path default Open

open-compass/opencompass#2615 · 1 assignee · Opened Aug 28, 2026
Feature request: EvalPort import/export for CustomDataset (adapter already built and tested) Open

open-compass/opencompass#2604 · 1 assignee · Opened Aug 21, 2026
CompassAcademic: 4 of 7 objective boards have an unseparated #1, two are exact ties broken by display order Open

open-compass/opencompass#2574 · 1 comment · 1 assignee · Opened Aug 1, 2026
[Bug] Qwen3.5 cannot run when using HuggingFacewithChatTemplate Open

open-compass/opencompass#2573 · 1 assignee · Opened Jul 31, 2026
MedFailBench integration proposal for clinical AI safety evaluation Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

open-compass/opencompass#2516 · 2 comments · Opened Jul 8, 2026
[Feature] DIK-structured Turkish syntactic dependency annotation dataset for structured linguistic evaluation Open

open-compass/opencompass#2508 · 1 comment · 1 assignee · Opened Jul 3, 2026
[Feature] 集成 ThinkCheck 作为 OpenCompass 的“推理健康度”评估维度 Open

open-compass/opencompass#2475 · 1 assignee · Opened Jun 8, 2026
[Feature] TruthfulQA mcq模式评测支持 Open

open-compass/opencompass#2427 · 1 comment · 1 assignee · Opened Mar 31, 2026
[Bug] [Feature] 评测框架代码过于耦合，失误得重新跑推理，文档中没有看到可以复用推理结果从新评测的命令 Open

open-compass/opencompass#2424 · 1 comment · 1 assignee · Opened Mar 30, 2026
[Feature] opencompass公开学术多模态榜单公布qwen3.5模型的结果 Open

open-compass/opencompass#2413 · 1 assignee · Opened Mar 13, 2026
[Feature] Question: where would a long horizon "tension crash test" TXT pack fit in OpenCompass? Open

open-compass/opencompass#2396 · 1 comment · 1 assignee · Opened Feb 6, 2026
Feature Request: Add Judge Calibration Diagnostics for LLM-as-Judge Evaluation Open

open-compass/opencompass#2392 · 1 comment · 1 assignee · Opened Feb 3, 2026
[Feature] Add another OpenAISDK model using prompt completion API Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

open-compass/opencompass#2386 · 2 comments · Opened Jan 14, 2026
[Bug] mmbench 文档不可访问 Open

open-compass/opencompass#2349 · 1 assignee · Opened Dec 8, 2025
[Feature] 使用飞书/QQ群来进行交流 Open

open-compass/opencompass#2348 · 1 assignee · Opened Dec 8, 2025
期望开通slack方便大家交流分享合作 Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

open-compass/opencompass#2341 · 1 comment · 3 reactions · Opened Dec 3, 2025
[Bug] 关于评测集社区-daily benchmark模块AI解读的问题 Open

open-compass/opencompass#2339 · 1 assignee · Opened Dec 1, 2025
[Bug] MedBench模型答案上传文件报错 Open

open-compass/opencompass#2338 · 1 assignee · Opened Dec 1, 2025
[Bug] 向medbench上传自测数据集失败 Open

open-compass/opencompass#2320 · 1 assignee · Opened Nov 22, 2025
[RFC] Add a board for local runner to track the states of all tasks Open

open-compass/opencompass#2317 · 2 comments · 1 assignee · Opened Nov 21, 2025
[Feature] Open

open-compass/opencompass#2315 · 1 assignee · Opened Nov 20, 2025
[Feature] support tablebench Open

open-compass/opencompass#2294 · 1 comment · 1 assignee · Opened Oct 23, 2025
[Feature] Addition of BioML-Bench Open

open-compass/opencompass#2279 · 1 comment · 1 assignee · Opened Sep 20, 2025
[Bug] MMLU 数据集在A100上评测运行日志显示某个批次的最后一个运行时就卡住不动，卡死了 Open

open-compass/opencompass#2278 · 1 comment · 1 assignee · Opened Sep 18, 2025
[Bug] How to evaluate infer results of multipl_e? Open

open-compass/opencompass#2265 · 1 assignee · Opened Sep 9, 2025
[Feature] Support pass@1 evaluation for multi predictions in MathEvaluator Open

open-compass/opencompass#2252 · 1 assignee · Opened Aug 28, 2025
[Feature] MindSpore framework support for OpenCompass Open

open-compass/opencompass#2243 · 1 comment · 1 assignee · Opened Aug 19, 2025
[Feature] 可视化工具 Open

open-compass/opencompass#2237 · 1 assignee · Opened Aug 12, 2025
[Bug] Unable to connect to Hugging Face's MultiPLE-Evaluator Open

open-compass/opencompass#2229 · 2 assignees · Opened Aug 7, 2025
[Bug] 官网的指导手册有更新嘛？和现在master的代码完全对不上，新版本的内容多久能更新，官网对应的版本是多少 Open

open-compass/opencompass#2197 · 3 comments · 1 reaction · 2 assignees · Opened Jul 10, 2025
[Bug] 无法提交PR，因为pre-commit-config.yaml中的update-dataset-suffix和update-dataset-suffix-pacakge钩子不兼容windows环境 Open

open-compass/opencompass#2164 · 2 assignees · Opened Jun 24, 2025
[Bug] OpenICLEvalTask does not combine results split by NumWorkerPartitioner Open

open-compass/opencompass#2163 · 1 assignee · Opened Jun 24, 2025
[Feature] 请问主观评测脚本支持用本地模型作为judge模型吗？ Open

open-compass/opencompass#2159 · 1 comment · 1 assignee · Opened Jun 18, 2025
[Feature] add AI/ML API as provider Open

open-compass/opencompass#2147 · 1 comment · 1 reaction · 1 assignee · Opened Jun 12, 2025
[Bug] qwen2.5-3b-Instruct点数对不上, oc测出来只有52，qwen2.5论文里面是72.7 Open

open-compass/opencompass#2141 · 1 assignee · Opened Jun 9, 2025
[Feature] Add mmlu-redux Open

open-compass/opencompass#2132 · 2 comments · 1 reaction · 1 assignee · Opened Jun 3, 2025
[Feature] 纯吐槽，Bug太多了，更新不及时！ Open

open-compass/opencompass#2119 · 9 comments · 6 reactions · 1 assignee · Opened May 26, 2025
[Bug] Qwen3评测分数低 Open

open-compass/opencompass#2093 · 8 comments · 1 assignee · Opened May 12, 2025
[Bug] I get aime2024 score 10 points on qwen 2.5 32b instruct Open

open-compass/opencompass#2042 · 2 comments · 1 assignee · Opened Apr 24, 2025
[Feature] Open

open-compass/opencompass#2040 · 1 comment · 1 assignee · Opened Apr 24, 2025
[Bug] Ceval_gen not compare for modelscope Open

open-compass/opencompass#2003 · 1 comment · 1 assignee · Opened Apr 4, 2025
[Bug] The ifeval evaluation on 541 cases returns unexpected low score. Open

open-compass/opencompass#1983 · 2 assignees · Opened Mar 29, 2025
[Feature] How about add bfcl evaluation to the repo Open

open-compass/opencompass#1981 · 2 comments · 1 reaction · 1 assignee · Opened Mar 28, 2025
[Bug] [Bug] LiveStemBench数据集评测时，推理结果正确。但是启动打分程序时报错 mmengine KeyError: '`cfg` or `default_args` must contain the key "type", but got {}\nNone' Open

open-compass/opencompass#1980 · 1 comment · 1 assignee · Opened Mar 28, 2025
[Bug] 多卡测试llama-3-8b-vllm，精度为0 Open

open-compass/opencompass#1979 · 2 comments · 1 assignee · Opened Mar 27, 2025
[Feature] 增加 benbench 评测集 Open

open-compass/opencompass#1975 · 1 comment · 1 assignee · Opened Mar 25, 2025

open-compass/opencompass

Get new issues in your inbox