open-compass / open-compass/VLMEvalKit
Adding LiveXiv - a live benchmark
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
Hi,
Thanks for the super important work of standardize evaluation!
I want to add my recent ICLR 2025 benchmark LiveXiv Which is a live and updated benchmark.
Is there a support for live benchmark or I need to add each version each time?
Thanks,
Nimrod
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names the LiveXiv dataset on Hugging Face but no repository files, tests, or entry points. Start by locating how VLMEvalKit registers benchmark datasets and handles versioned data, then determine whether a live benchmark can be supported without adding each version separately. Done means the supported workflow is clearly defined and LiveXiv integration is covered by the relevant validation or evaluation checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100