open-compass / open-compass/VLMEvalKit
ai2d数据集结果不正常
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
我在测试Qwen3-VL-4B-Instruct，执行
CUDA_VISIBLE_DEVICES="5,6" python VLMEvalKit/run.py --model "$MODEL_NAME" --work-dir "$OUTPUT_PATH" --data "AI2D_TEST"
测试结果不正常，太低了
------------------------- -------------------
split none
Overall 0.19203367875647667
atomStructure 0.125
eclipses 0.22580645161290322
faultsEarthquakes 0.03571428571428571
foodChainsWebs 0.20395869191049915
lifeCycles 0.1366906474820144
moonPhaseEquinox 0.18050541516245489
partsOfA 0.26078028747433263
partsOfTheEarth 0.23076923076923078
photosynthesisRespiration 0.11392405063291139
rockCycle 0.14925373134328357
rockStrata 0.07317073170731707
solarSystem 0.4166666666666667
typesOf 0.16326530612244897
volcano 0.3125
waterCNPCycle 0.06818181818181818
------------------------- -------------------
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported VLMEvalKit/run.py command and the AI2D_TEST evaluation output. Reproduce the Qwen3-VL-4B-Instruct run, inspect the AI2D evaluation path and inputs, and identify why the reported accuracy is unexpectedly low; done means a confirmed cause and corrected or explained result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100