mlcommons / mlcommons/inference
Tokens per sample upper limit for GPTJ
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
Is there any reason why we have an accuracy upper limit for LLAMA2 Tokens per sample but not for GPT-J? It's good to document this reason for users.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tools/submission/submission_checker.py around line 109 and compare the LLAMA2 token limit with the GPT-J validation path. Determine the reason for the difference and identify where that rationale should be documented for users. Done means the behavior and its rationale are clearly recorded, with any related checker change confirmed by maintainers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100