open-compass / open-compass/VLMEvalKit
Qwen3-VL-4B-Instruct模型评估ocrbench结果和官方公布结果不一致
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
问题1:评测结果不一致:
本地评测出来结果为87,详细信息如下:
官方公布的结果为88.1,相差一个点,
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 469: The evaluation of model Qwen3-VL-4B-Instruct x dataset OCRBench has finished!
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 470: Evaluation Results:
[2025-12-30 11:26:43] INFO - RUN - run.py: main - 472:
{
"Text Recognition": 265,
"Scene Text-centric VQA": 180,
"Doc-oriented VQA": 184,
"Key Information Extraction": 181,
"Handwritten Mathematical Expression Recognition": 60,
"Final Score": 870,
"Final Score Norm": 87.0
}
评测配置如下:
"Qwen3-VL-4B-Instruct": partial(
Qwen3VLChat,
# model_path="Qwen/Qwen3-VL-4B-Instruct",
model_path="./model/Qwen3-VL-4B-Instruct",
use_custom_prompt=False,
use_vllm=False,
temperature=0.7,
max_new_tokens=200,
repetition_penalty=1.0,
presence_penalty=1.5,
top_p=0.8,
top_k=20
),
问题2: qwen3vl的在mmstar之类的选择题上选项解释非常多,导致评测结果很长。默认的配置( use_custom_prompt=False)也没有使用qwen3vl自己的prompt实现(实测use_custom_prompt=True之后ocrbench下降到85)。是否有合适的配置?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with the run.py entries at lines 469–472 and the Qwen3VLChat configuration shown. Reproduce OCRBench and MMStar using the listed generation settings, then compare the local scores and answer lengths with the reported official results. Done means identifying the configuration or prompt cause and documenting a verified setup with expected results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100