open-compass / open-compass/VLMEvalKit
model：llava-1.5-7b，data：mmebench
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
我的模型是llava-1.5-7b的模型，我想要评测mmbench，我分别测试了cn与en。我得到的结果分别为：mmbench-en：62.8（未调用api）；63.12（调用api）；llava官方结果为64.3； mmbench-cn：22.4（未调用api）；44.07（调用api）；llava官方结果为58.3； 想问问您之前有没有用vlevalkit测过llava呀？
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported VLMEvalKit results for llava-1.5-7b on MMBench Chinese and English, both with and without API calls. Compare the outputs with the stated official LLaVA scores and determine whether the discrepancy is caused by evaluation configuration or an implementation issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100