huggingface / huggingface/diffusers

Classifier free guidance(CFG) on different prediction types and karras style schedulers

Open
#8,617 3 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Python
Stars
34.5k
Forks
7.3k
Avg merge
3d 3h
Merged PRs (30d)
91

Description

There is banch of repos [1](https://github.com/duxiaodan/intrinsic-lora/blob/b69770c53035b936eae3132354afb1e14d2043ea/rescale_cfg_pipeline_forward.py#L259), [2](https://github.com/Amblyopius/Stable-Diffusion-ONNX-FP16/blob/55d050957f70c90f14aed13d9854f7aa59ff1f70/pipeline_onnx_stable_diffusion_instruct_pix2pix.py#L361), [3](https://github.com/scnuhealthy/video_try_on/blob/4b72bb32c59d37ed6b16a61f6906d014bc511e69/video_models/video_pipeline_mae_guided.py#L308) there autors take into account type of noise scheduler and for karras like change model output before cfg calculation:
```
# Hack:
# For karras style schedulers the model does classifer free guidance using the
# predicted_original_sample instead of the noise_pred. So we need to compute the
# predicted_original_sample here if we are using a karras style scheduler.
if scheduler_is_in_sigma_space:
step_index = (self.scheduler.timesteps == t).nonzero()[0].item()
sigma = self.scheduler.sigmas[step_index]
noise_pred = latent_model_input - sigma * noise_pred

# perform guidance
if self.do_classifier_free_guidance:
noise_pred_text_image, noise_pred_text = noise_pred.chunk(2)
noise_pred = (
noise_pred_text
+ self.image_guidance_scale * (noise_pred_text_image - noise_pred_text)
)

# Hack:
# For karras style schedulers the model does classifer free guidance using the
# predicted_original_sample instead of the noise_pred. But the scheduler.step function
# expects the noise_pred and computes the predicted_original_sample internally. So we
# need to overwrite the noise_pred here such that the value of the computed
# predicted_original_sample is correct.
if scheduler_is_in_sigma_space:
noise_pred = (noise_pred - latents) / (-sigma)
```

Firstly, I don’t know is this step so important and where find the original idea for cfg in karras like schedulers. Secondly, I wondered whether a similar effect should be considered for different types of network output (x0, eps, v). I will be very grateful for any investigaion on this topic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.