kohya-ss / kohya-ss/sd-scripts

Bug+Fix: Broken sampling with DDIM during training

Open
#1,432 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I was wondering why the sampling during training produced such bad results and after many hours of debugging, I found the issue.
The `clip_sample` value in the scheduler configuration [must be False](https://huggingface.co/CompVis/stable-diffusion-v1-3/discussions/4)!

Currently, it is explicitly set to True in `get_my_scheduler`, then set to False by the StableDiffusionPipeline, which also produces two warnings. Unfortunately, this safety net doesn't work if the Scheduler is re-assigned again later, which happens when the sampling occurs.

So, long story short, I suggest the following change.
In **sd-scripts/library/train_util.py**, in the function **get_my_scheduler**, replace the code:

https://github.com/kohya-ss/sd-scripts/blob/bfb352bc433326a77aca3124248331eb60c49e8c/library/train_util.py#L5031-L5034

with
```python
# steps_offset must be 1 according to StableDiffusionPipeline in the diffusers library
if hasattr(scheduler.config, "steps_offset") and scheduler.config.steps_offset != 1:
scheduler.config.steps_offset = 1

# clip_sample must be False according to the StableDiffusionPipeline in the diffusers library,
# will mess up the sampling process for ddim if set to True!
if hasattr(scheduler.config, "clip_sample") and scheduler.config.clip_sample is True:
scheduler.config.clip_sample = False
```
This change will also prevent the two related deprecation warnings from the StableDiffusionPipeline and improve compatibility with the current version of the diffusers library.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in library/train_util.py, in get_my_scheduler, and inspect the existing scheduler configuration handling around the linked lines. Verify the DDIM training-sampling path and the StableDiffusionPipeline compatibility behavior. Done means steps_offset is normalized to 1, clip_sample is set to False when needed, and the two related deprecation warnings are no longer produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.