open-compass / open-compass/VLMEvalKit
Which JudgeModel is it?
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
Which evaluation sets are scored based on rules?
Which evaluation sets are scored based on GPT, and which specific version of GPT is it?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the evaluation-set scoring definitions and the JudgeModel references in the repository. Trace which sets use rule-based scoring versus GPT-based judging, record the GPT version for each, and document the results in the relevant project documentation; the issue does not name a file or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100