open-compass / open-compass/VLMEvalKit
How to use a locally deployed judge model, such as Qwen3-8B?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
I followed the steps in the provided link to deploy Qwen3-8B locally as a judge model. However, when evaluating the MMBench_DEV_EN_V11 dataset, an error was thrown at line 263 in vlmeval/dataset/image_mcq.py:
assert model in ['chatgpt-0125', 'exact_matching', 'gpt-4-0125']
I noticed that many datasets only support chatgpt-0125 and gpt-4-0125 as judge models. If that’s the case, what is the purpose of the local judge model deployment tutorial in Quickstart.md? How can I actually use a locally deployed judge model?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the local judge model deployment section in Quickstart.md, then inspect vlmeval/dataset/image_mcq.py around line 263 and the MMBench_DEV_EN_V11 evaluation path. Confirm which judge models each dataset accepts and whether the documented local deployment path covers them. Done means the supported workflow and any dataset limitations are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100