huggingface / huggingface/diffusers
Classifier free guidance(CFG) on different prediction types and karras style schedulers
- Dominant language
- Python
- Stars
- 34.5k
- Forks
- 7.3k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 91
Description
There is banch of repos [1](https://github.com/duxiaodan/intrinsic-lora/blob/b69770c53035b936eae3132354afb1e14d2043ea/rescale_cfg_pipeline_forward.py#L259), [2](https://github.com/Amblyopius/Stable-Diffusion-ONNX-FP16/blob/55d050957f70c90f14aed13d9854f7aa59ff1f70/pipeline_onnx_stable_diffusion_instruct_pix2pix.py#L361), [3](https://github.com/scnuhealthy/video_try_on/blob/4b72bb32c59d37ed6b16a61f6906d014bc511e69/video_models/video_pipeline_mae_guided.py#L308) there autors take into account type of noise scheduler and for karras like change model output before cfg calculation:
```
# Hack:
# For karras style schedulers the model does classifer free guidance using the
# predicted_original_sample instead of the noise_pred. So we need to compute the
# predicted_original_sample here if we are using a karras style scheduler.
if scheduler_is_in_sigma_space:
step_index = (self.scheduler.timesteps == t).nonzero()[0].item()
sigma = self.scheduler.sigmas[step_index]
noise_pred = latent_model_input - sigma * noise_pred
# perform guidance
if self.do_classifier_free_guidance:
noise_pred_text_image, noise_pred_text = noise_pred.chunk(2)
noise_pred = (
noise_pred_text
+ self.image_guidance_scale * (noise_pred_text_image - noise_pred_text)
)
# Hack:
# For karras style schedulers the model does classifer free guidance using the
# predicted_original_sample instead of the noise_pred. But the scheduler.step function
# expects the noise_pred and computes the predicted_original_sample internally. So we
# need to overwrite the noise_pred here such that the value of the computed
# predicted_original_sample is correct.
if scheduler_is_in_sigma_space:
noise_pred = (noise_pred - latents) / (-sigma)
```
Firstly, I don’t know is this step so important and where find the original idea for cfg in karras like schedulers. Secondly, I wondered whether a similar effect should be considered for different types of network output (x0, eps, v). I will be very grateful for any investigaion on this topic.
Contributor guide
Assessment
This issue has not been assessed yet.