kohya-ss / kohya-ss/sd-scripts
sd3 train,loss is nan
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
when I use this command, the loss is nan,how to solve this problem? Thanks for your help
the datasets is small,just 290 images, but loss is nan，I try to set the mixed_precision=bf16 and t5xxl_dtype =bf16,but these settings don't work ,the loss is also nan
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the training command and settings shown in the attached images, including mixed_precision, t5xxl_dtype, and the 290-image dataset configuration. Reproduce the SD3 training run and determine why the loss becomes NaN; done means identifying a reproducible cause and confirming that the loss remains finite with the corrected setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100