open-compass / open-compass/VLMEvalKit
Qwen2.5-VL-3B-Instruct在很多评测数据集的指标与官方榜单差距较大
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
torch版本2.5.1
transformers版本4.49.0.dev0
judge模型gpt-4o-turble
generation_config:
top_p=0.001,
top_k=1,
temperature=0.01,
repetition_penalty=1.0,
自测结果：
AI2D: 0.7836
DynaMath (worst case overall): 0.067
HallusionBench: 42.49
MMU(val): 0.48
MMStar: 0.544
官方结果：
AI2D: 0.814
DynaMath (worst case overall): 0.132
HallusionBench: 46.6
MMU(val): 0.512
MMStar: 0.563
在其他一些数据集上我测的结果比官方榜单高，总体avg差距不是特别大。
但是上述的DynaMath、HallusionBench、MMU(val)和榜单差距有点太大了。不知道是什么原因造成的呢？
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named in the report. Start by reproducing the listed benchmark results with torch 2.5.1, transformers 4.49.0.dev0, the stated judge model, and generation settings, then compare them with the official evaluation setup. Done means identifying the cause of the metric differences and documenting a reproducible configuration or correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100