open-compass / open-compass/VLMEvalKit
ScienceQA_TEST的评测怀疑与官方的测评集不对齐!
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
根据ScienceQA论文和官网 https://scienceqa.github.io/index.html#home,训练集、验证集、测试集是3:1:1,测试集应该有4241张图片。
modelscope里面的
https://modelscope.cn/datasets/swift/ScienceQA 的文档写的数据量是正确的,
执行具体代码
from modelscope.msdatasets import MsDataset
ds = MsDataset.load('AI-ModelScope/ScienceQA', subset_name='default', split='test')
len(ds)==4241
也是正确的,是True。
但是VLMEvalKit的评测结果是错的,实测发现 只有2017条数据。
这是我的运行命令
swift eval \
--eval_backend VLMEvalKit \
--infer_backend pt \
--eval_dataset ScienceQA_TEST \
--model QwenQwen3-VL-4B-Instruct
结果文件在
./eval_output/vlmeval/时间/Qwen3-VL-4B-Instruct/Qwen3-VL-4B-Instruct_ScienceQA_TEST_openai_result.xlsx
中,这个只有一个表格,只有2017条数据。
看起来VLMEvalKit的 ScienceQA_TEST 没有正确对齐官方数据,请问这里面是有什么原因吗?是不是有些数据被特意筛选呢?还是存在一些技术问题?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the shown swift eval command and compare the ScienceQA_TEST result count with the official test size of 4241 and the ModelScope split. Trace how VLMEvalKit loads ScienceQA_TEST and determine whether the 2017-item result is intentional filtering or a loading mismatch; done means the cause is confirmed and the dataset behavior is corrected or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100