open-compass / open-compass/VLMEvalKit
difference sample number with HallusionBench's paper
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
I try to evaluate models on HallusionBench.
The number of hallusionbench in lmevalkit is 951, which is 1129 in HallusionBench's huggingface.
I wonder why they have different number.
What's more, I get only about 30 score of seed1.5vl on hallusionbench, which is far from 60.3 in the seed1.5vl paper.
Could someone get the similar result with me?
Thanks.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the HallusionBench sample count in VLMEvalKit with the 1129 samples on HallusionBench's Hugging Face version. Then reproduce the Seed1.5VL evaluation and compare the roughly 30 score with the 60.3 reported in its paper. Done means the dataset and score discrepancies are explained or reproducibly confirmed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100