open-compass / open-compass/VLMEvalKit
Support judge model for MAC&ShortVQA Question
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
Support judge model for MCQ&ShortVQA Question
https://huggingface.co/IAAR-Shanghai/xVerify-9B-C 。
支持一些 MAC&ShortVQA 问题的 judge model；
在实际评测中，为了测试一个 MCQ / VQA 数据集，会考虑非常多的 PROMPT工程以及后处理。但越面向用户的模型，输出的多样性越高。后处理的复杂度越高，写起来也越累，错误率还特别高，这让后处理越来越复杂。
在查看 ChartQA 测试的结果时，发现 规则提取的答案错误率非常高，尝试了 gpt4omini/deepseekv3 评价，发现错误率还是比较高，还需要设计 system_prompt 以及 few-shot。 当我使用 xVerify-9B-C 事情变得简单，正确率也高很多。
人工check所有的评测结果后，错误率大约是：
规则的错误率： 5% ～6 %
gpt4omini 错误率: 2% ~ 3%
xverify 的错误率： 0.5% ~ 0.6%
vllm 单卡的服务大约 40QPS （eval 40条数据每秒）
可能和这些模型训过 ChartQA 相关数据有影响，但后续使用在countbench 等数据集上时，表现非常好，很难发现误判。
最大的好处时 较低成本下，可以忽视掉 prompt工程和后处理的代码！！！
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing MCQ, ShortVQA, and ChartQA evaluation entry points and any judge-model integrations. Review how xVerify-9B-C is served with vLLM, then determine the project interface and tests needed for the model; done means MCQ and ShortVQA evaluation can use it reliably without bespoke prompt or answer post-processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100