open-compass / open-compass/VLMEvalKit
很多模型使用vlmevalkit评测不同的数据集时效果都有较大的差距
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
mPLUG code with paper榜单上coco数据集的BLEU-4为46.5，CIDER为155.1
实际我跑的BLEU-4为33.4，CIDER为95.61，结果如下：
（另外，评测指标能不能尽量完整一些，补全METEOR和ROUGE-L）
还有minicpmv2.6在mme-video上面的结果也较差，这个我在其他的issue当中也看到了类似的问题，这里再次提出，希望能够得到解决。
以下是我的验证脚本，使用的swift。
CUDA_VISIBLE_DEVICES=0,1,2,3
swift eval
--adapters /share/user/output/mPLUG-Owl3-7B-240728/v1-20250421-170456/checkpoint-4405
--eval_backend VLMEvalKit
--infer_backend pt
--eval_dataset COCO_VAL
--temperature 0
感谢您和您团队的工作，希望能够尽快完善！
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied swift eval command, the VLMEvalKit evaluation backend, and the COCO_VAL dataset entry point. Compare the reported mPLUG results with the reference metrics, then inspect the evaluation path for MiniCPMV2.6 on MME-Video. Done means the discrepancies are explained or corrected and METEOR and ROUGE-L support is clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100