huggingface / huggingface/diffusers

[SD3 ControlNet] bug in pipeline 'controlnet_pooled_projections'

Aperta
#9,686 11 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

### Describe the bug

Hi,

I think I found an issue that causes a misalignment between training and inference in SD3 ControlNet.

https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet.py#L977

I think the if-else block starting there is not correct. It should be
```python
if controlnet_pooled_projections is None and pooled_prompt_embeds is None:
controlnet_pooled_projections = torch.zeros_like(pooled_prompt_embeds)
elif controlnet_pooled_projections is None:
controlnet_pooled_projections = pooled_prompt_embeds
```
Given that in training, the pooled_prompt_embeds are fed to the model:
https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/examples/controlnet/train_controlnet_sd3.py#L1293

Additionally, I am wondering if this line:
https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/examples/controlnet/train_controlnet_sd3.py#L1287
Should be aligned with this line:
https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/examples/controlnet/train_controlnet_sd3.py#L1257
This seems to be the more sensible approach, but will probably not make much difference since the ControlNet can also learn the shift. It might speed up convergence *slightly*.

Best,
Tobias

### Reproduction

Train an SD3 ControlNet and during log_validation it will be executed.

### Logs

_No response_

### System Info

diffusers==0.30.3

### Who can help?

@yiyixuxu @sayakpaul

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.