open-compass / open-compass/VLMEvalKit
关于qwen2.5omni的性能
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
你好，我使用VLMEvalKit 对qwen2.5-omni进行评估时发现在几个数据集上的性能和论文中的差距较大，请问这是为什么，下面是我评测的几个数据集性能
| 数据集 | 论文 | 评估 | 命令 |
|---|---|---|---|
| MMMU_val | 59.2 | 51.6 | python run.py --data MMMU_DEV_VAL --model Qwen2.5-Omni-7B |
| MMMU_pro | 36.6 | 31 | python run.py --data MMMU_Pro_10c MMMU_Pro_V --model Qwen2.5-Omni-7B |
| MMStar | 64.0 | 60.6 | python run.py --data MMStar --model Qwen2.5-Omni-7B |
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported VLMEvalKit commands in the issue and compare their MMMU_val, MMMU_pro, and MMStar settings with the paper's evaluation setup for Qwen2.5-Omni-7B. Check whether the dataset names, model configuration, and scoring procedure match. Done means identifying the source of the performance gap or documenting why the results cannot be directly compared.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100