NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Seeing nan loss when setting packed_sequence_size > 0
Open
A pull request for this has already been merged.
- #792 by @adil-a — merged
bug
external
qa_rcca_done
waiting-on-customer
x-npci
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Describe the bug
Running the train_ft.py for Mistral 24B model with packed_sequence_size = 4096, results in nan as loss.
Edit: setting split_across_packs=False bypasses the issue.
Steps/Code to reproduce bug
change model to mistralai/Mistral-Small-24B-Instruct-2501 and packed_sequence_size = 4096
Expected behavior
Model training with correct loss values
Additional context
Using NeMo 25.09.00 container for running this.
model_distribution.py
LlamaConfig.yaml
training_20251028_112815.log
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.