NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Nemotron-Nano-9b-V2 has low full-weights SFT accuracy
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Describe the bug
Nemotron-Nano-9b-V2 full weights SFT accuracy is really low. The rouge score is 0.07. However, the older AutoModel training scripts give a higher rouge score (0.23) for the same hyper-parameters.
Steps/Code to reproduce bug
torchrun --nproc-per-node=8 /opt/Automodel/examples/llm_finetune/finetune.py --config config.yaml
Config.yaml - config.yaml.rtf
Dataset is attached.
email-composition-train.tar.gz
Expected behavior
Base model rouge score is at least 0.23. Full weights SFT should give at least 0.23.
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the provided torchrun command, config.yaml, and email-composition dataset, starting at examples/llm_finetune/finetune.py. Compare this path with the older AutoModel training scripts using the same hyper-parameters. Done means full-weights SFT reaches the expected ROUGE score of at least 0.23.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100