modelscope / modelscope/DiffSynth-Studio
NaN Loss after Forward Pass on Step 4 when Full Fine-Tuning Wan 2.2 5B
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
Hi, I am currently full fine-tuning the Wan 2.2 5B model, but the loss becomes NaN right after the forward process at the 4th training step every time.
I have verified that the issue is not related to specific data samples — each training run uses a different random seed, so the order of loaded input videos changes every time, yet the NaN loss still consistently occurs at the same step.
What could be the possible reasons for this NaN loss?
Thanks！
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing full fine-tuning of the Wan 2.2 5B model and inspect the forward pass and loss at training step 4 across the available random seeds. Since the issue names no files or tests, trace the training entry point and record the cause and a verified resolution for the NaN loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100