open-compass / open-compass/VLMEvalKit
评测MMBench数据集发现部分问题判别错误
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
评测推理产生的文件为：Qwen3.5-27B_MMBench_DEV_CN_V11.xlsx
判别产生的文件为：Qwen3.5-27B_MMBench_DEV_CN_V11_GLM4.7_result.xlsx
eval命令：python run.py --data MMBench_DEV_CN_V11 --model Qwen3.5-27B --mode eval --reuse --judge GLM4.7 --verbose --judge-args '{"temperature":0, "chat_template_kwargs":{"enable_thinking":false}}'
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached evaluation and judgment spreadsheets, then reproduce the issue using the documented python run.py command for MMBench_DEV_CN_V11 with GLM4.7. Compare the generated judgments with the evaluation output to identify which questions are misclassified. The issue is complete when the cause and required correction are established and the affected cases are validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100