Lightning-AI / Lightning-AI/litgpt
[suggestion] Lit is missing Alpaca_Eval and MT benchmark
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Lit-gpt has superb libraries for instruction tuning. But instruction tuned models are popularly never used for eval using lm eval harness.
MT benchmark: https://lmsys.org/blog/2023-06-22-leaderboard/
alpaca eval: https://github.com/tatsu-lab/alpaca_eval
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing LitGPT's existing evaluation entry points, then compare their capabilities with the lm eval harness and the linked Alpaca Eval project. The MT benchmark and Alpaca Eval references define the requested evaluation targets; done would mean a decided, documented way for LitGPT models to run both evaluations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100