open-compass / open-compass/VLMEvalKit
Unable to reproduce Intern-S1-mini on XLRS-Bench-lite
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
Hi，
I used the following command to reproduce the results of the Intern-S1-mini model on XLRS-Bench-lite, and only achieved a score of 14.51, which is significantly different from the published score of 51.63 for Intern-S1-mini.
python run.py --data XLRS-Bench-lite --model Intern-S1-mini --verbose
Upon troubleshooting, I found that the class method XLRSBench.evaluate() of the XLRS-Bench-lite dataset only uses rule-based methods to extract answers. Since this class inherits from the base class ImageMCQDataset, could you modify the source code to add judge_model?
Best regards
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the reported command in run.py and reading XLRSBench.evaluate(), along with the inherited ImageMCQDataset implementation. Confirm how XLRS-Bench-lite currently extracts answers and how judge_model is handled elsewhere; done means evaluation supports the requested judging path and the reproduction score can be checked against the published result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100