NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Pretrain LLama 7b on 200B tokens
Open
@HuiyingLi is already working on this.
Since Dec 2, 2025.
enhancement
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Is your feature request related to a problem? Please describe.
Goals:
- Validate NeMo Automodel pretraining on a llama 7b for 200B tokens
- Add loss & validation scores to docs.
- Validation on: arc_challenge, arc_easy, boolq, copa, hellaswag, obqa, piqa, siqa, winogrande, nq, tqa
Describe the solution you'd like
Documented pretrain run on large dataset
Describe alternatives you've considered
N/A
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.