huggingface / huggingface/diffusers

Classifier free guidance(CFG) on different prediction types and karras style schedulers

Aperta
#8,617 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

There is banch of repos [1](https://github.com/duxiaodan/intrinsic-lora/blob/b69770c53035b936eae3132354afb1e14d2043ea/rescale_cfg_pipeline_forward.py#L259), [2](https://github.com/Amblyopius/Stable-Diffusion-ONNX-FP16/blob/55d050957f70c90f14aed13d9854f7aa59ff1f70/pipeline_onnx_stable_diffusion_instruct_pix2pix.py#L361), [3](https://github.com/scnuhealthy/video_try_on/blob/4b72bb32c59d37ed6b16a61f6906d014bc511e69/video_models/video_pipeline_mae_guided.py#L308) there autors take into account type of noise scheduler and for karras like change model output before cfg calculation:
```
# Hack:
# For karras style schedulers the model does classifer free guidance using the
# predicted_original_sample instead of the noise_pred. So we need to compute the
# predicted_original_sample here if we are using a karras style scheduler.
if scheduler_is_in_sigma_space:
step_index = (self.scheduler.timesteps == t).nonzero()[0].item()
sigma = self.scheduler.sigmas[step_index]
noise_pred = latent_model_input - sigma * noise_pred

# perform guidance
if self.do_classifier_free_guidance:
noise_pred_text_image, noise_pred_text = noise_pred.chunk(2)
noise_pred = (
noise_pred_text
+ self.image_guidance_scale * (noise_pred_text_image - noise_pred_text)
)

# Hack:
# For karras style schedulers the model does classifer free guidance using the
# predicted_original_sample instead of the noise_pred. But the scheduler.step function
# expects the noise_pred and computes the predicted_original_sample internally. So we
# need to overwrite the noise_pred here such that the value of the computed
# predicted_original_sample is correct.
if scheduler_is_in_sigma_space:
noise_pred = (noise_pred - latents) / (-sigma)
```

Firstly, I don’t know is this step so important and where find the original idea for cfg in karras like schedulers. Secondly, I wondered whether a similar effect should be considered for different types of network output (x0, eps, v). I will be very grateful for any investigaion on this topic.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start by comparing the three linked pipeline implementations, focusing on the scheduler_is_in_sigma_space branches and the classifier-free guidance calculation. Trace how scheduler.step consumes noise_pred and how the examples handle predicted-original-sample values, then investigate whether the same concern applies to x0, eps, and v prediction types. Done means documenting the correct behavior and identifying an agreed implementation or test scope.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.