mlfoundations / mlfoundations/evalchemy
A suggestion for evalchemy about LiveCodeBench
Open
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 610
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
Official repo of LiveCodeBench support different versions of data for evaluation, but in your repo there is only release_v2. It would be better to add some args for the user to choose.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Compare evalchemy's current release_v2 handling with the official LiveCodeBench repository's supported data versions, then locate the evaluation entry point and existing argument parsing. Done means users can select an available LiveCodeBench data version instead of being limited to release_v2.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100