Lightning-AI / Lightning-AI/litgpt
Evaluation with triviaqa
Open
Nobody has claimed this yet.
bug
evaluation
waiting on author
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Triviaqa gives 0 acc while when I tried evaluating it with LLaMA2-7B. I am curious why is this happening?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named. Start by locating the TriviaQA evaluation path and reproducing the reported zero accuracy with LLaMA2-7B; done means identifying and documenting the cause or confirming the evaluation behaves correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100