open-compass / open-compass/VLMEvalKit

ScienceQA_TEST的评测怀疑与官方的测评集不对齐!

Open
#1,300 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
1d 10h
Merged PRs (30d)
17

Description

根据ScienceQA论文和官网 https://scienceqa.github.io/index.html#home,训练集、验证集、测试集是3:1:1,测试集应该有4241张图片。

modelscope里面的
https://modelscope.cn/datasets/swift/ScienceQA 的文档写的数据量是正确的,

执行具体代码

from modelscope.msdatasets import MsDataset
ds =  MsDataset.load('AI-ModelScope/ScienceQA', subset_name='default', split='test')
len(ds)==4241

也是正确的,是True。

但是VLMEvalKit的评测结果是错的,实测发现 只有2017条数据。

这是我的运行命令

swift eval \
    --eval_backend VLMEvalKit \
    --infer_backend pt \
    --eval_dataset ScienceQA_TEST \
    --model QwenQwen3-VL-4B-Instruct

结果文件在
./eval_output/vlmeval/时间/Qwen3-VL-4B-Instruct/Qwen3-VL-4B-Instruct_ScienceQA_TEST_openai_result.xlsx
中,这个只有一个表格,只有2017条数据。

看起来VLMEvalKit的 ScienceQA_TEST 没有正确对齐官方数据,请问这里面是有什么原因吗?是不是有些数据被特意筛选呢?还是存在一些技术问题?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the shown swift eval command and compare the ScienceQA_TEST result count with the official test size of 4241 and the ModelScope split. Trace how VLMEvalKit loads ScienceQA_TEST and determine whether the 2017-item result is intentional filtering or a loading mismatch; done means the cause is confirmed and the dataset behavior is corrected or documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.