open-compass / open-compass/VLMEvalKit
是否支持数据集指定索引的评测？
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
测试一个数据集的时候，有些问题想要重复测试一下，是否能将这些问题抽出来二次评测
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing how VLMEvalKit loads datasets and selects samples for evaluation, then clarify the intended way to specify indices and whether results should remain comparable to full-dataset runs. Done means selected questions can be evaluated independently and the behavior is covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100