kohya-ss / kohya-ss/sd-scripts
Bug+Fix: Broken sampling with DDIM during training
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I was wondering why the sampling during training produced such bad results and after many hours of debugging, I found the issue.
The `clip_sample` value in the scheduler configuration [must be False](https://huggingface.co/CompVis/stable-diffusion-v1-3/discussions/4)!
Currently, it is explicitly set to True in `get_my_scheduler`, then set to False by the StableDiffusionPipeline, which also produces two warnings. Unfortunately, this safety net doesn't work if the Scheduler is re-assigned again later, which happens when the sampling occurs.
So, long story short, I suggest the following change.
In **sd-scripts/library/train_util.py**, in the function **get_my_scheduler**, replace the code:
https://github.com/kohya-ss/sd-scripts/blob/bfb352bc433326a77aca3124248331eb60c49e8c/library/train_util.py#L5031-L5034
with
```python
# steps_offset must be 1 according to StableDiffusionPipeline in the diffusers library
if hasattr(scheduler.config, "steps_offset") and scheduler.config.steps_offset != 1:
scheduler.config.steps_offset = 1
# clip_sample must be False according to the StableDiffusionPipeline in the diffusers library,
# will mess up the sampling process for ddim if set to True!
if hasattr(scheduler.config, "clip_sample") and scheduler.config.clip_sample is True:
scheduler.config.clip_sample = False
```
This change will also prevent the two related deprecation warnings from the StableDiffusionPipeline and improve compatibility with the current version of the diffusers library.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in library/train_util.py, in get_my_scheduler, and inspect the existing scheduler configuration handling around the linked lines. Verify the DDIM training-sampling path and the StableDiffusionPipeline compatibility behavior. Done means steps_offset is normalized to 1, clip_sample is set to False when needed, and the two related deprecation warnings are no longer produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100