open-compass / open-compass/VLMEvalKit
mvbench bug
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
这里应该是 data.loc[data['index'] == idx, 'score'] = int(check_ans_with_model ?
跑mvbench前十个子集的时候发现出现NaN的类别，看着像这行的问题
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at vlmeval/dataset/mvbench.py around line 410 and inspect the score assignment referenced in the issue. Reproduce the run for the first ten MVBench subsets, verify which categories produce NaN, and confirm that the corrected assignment produces valid scores for them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100