Lightning-AI / Lightning-AI/litgpt
Code evaluation using bigcode-evaluation-harness framework
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Code evaluation task/benchmark such as HumanEval and MBPP are missing from **lm-evaluation-harness**, but are present and maintained in **bigcode-evaluation-harness**.
https://github.com/bigcode-project/bigcode-evaluation-harness
Since, we would need to parse tasks and check if they are in lm-evaluation-harness or bigcode-evaluation-harness, I propose to keep `litgpt evaluate` but add argument `--framework "lm-evaluation-harness"` (default if not specified) or `--framework "bigcode-evaluation-harness"`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the `litgpt evaluate` entry point and inspect how tasks are currently parsed and dispatched to lm-evaluation-harness. Determine where a `--framework` choice can route tasks to bigcode-evaluation-harness; done means HumanEval and MBPP can be evaluated through the proposed framework without breaking the existing default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100