open-compass / open-compass/VLMEvalKit
Discrepancy in CountBenchQA Data Count Compared to Original Paper
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
Hello, Thank you for your great work on this repository. I’ve been using it recently and noticed something I’m unsure about. In the CountBenchQA dataset, I found that there are only 487 entries, while the original paper mentions 540.
I’m wondering if I might be misunderstanding something or if there’s a step I may have missed. Could you please clarify this discrepancy?
Thanks in advance for your help!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue mentions CountBenchQA and the original paper but no repository files or tests. Start by verifying how the dataset’s 487 entries compare with the paper’s 540; done means documenting the cause or confirming what step or subset accounts for the difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100