open-compass / open-compass/VLMEvalKit

Qwen3-VL-4B-Instruct模型评估ocrbench结果和官方公布结果不一致

Open
#1,389 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
2d 27m
Merged PRs (30d)
18

Description

问题1:评测结果不一致:
本地评测出来结果为87,详细信息如下:
官方公布的结果为88.1,相差一个点,
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 469: The evaluation of model Qwen3-VL-4B-Instruct x dataset OCRBench has finished!
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 470: Evaluation Results:
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 472:
{
"Text Recognition": 265,
"Scene Text-centric VQA": 180,
"Doc-oriented VQA": 184,
"Key Information Extraction": 181,
"Handwritten Mathematical Expression Recognition": 60,
"Final Score": 870,
"Final Score Norm": 87.0
}

评测配置如下:
"Qwen3-VL-4B-Instruct": partial(
Qwen3VLChat,
# model_path="Qwen/Qwen3-VL-4B-Instruct",
model_path="./model/Qwen3-VL-4B-Instruct",
use_custom_prompt=False,
use_vllm=False,
temperature=0.7,
max_new_tokens=200,
repetition_penalty=1.0,
presence_penalty=1.5,
top_p=0.8,
top_k=20
),

问题2: qwen3vl的在mmstar之类的选择题上选项解释非常多,导致评测结果很长。默认的配置( use_custom_prompt=False)也没有使用qwen3vl自己的prompt实现(实测use_custom_prompt=True之后ocrbench下降到85)。是否有合适的配置?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the run.py entries at lines 469–472 and the Qwen3VLChat configuration shown. Reproduce OCRBench and MMStar using the listed generation settings, then compare the local scores and answer lengths with the reported official results. Done means identifying the configuration or prompt cause and documenting a verified setup with expected results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.