huggingface / huggingface/diffusers
[SD3 ControlNet] bug in pipeline 'controlnet_pooled_projections'
- Lenguaje dominante
- Python
- Estrellas
- 34.5k
- Forks
- 7.3k
- Merge medio
- 3 d 3 h
- PR fusionados (30 d)
- 91
Descripción
### Describe the bug
Hi,
I think I found an issue that causes a misalignment between training and inference in SD3 ControlNet.
https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet.py#L977
I think the if-else block starting there is not correct. It should be
```python
if controlnet_pooled_projections is None and pooled_prompt_embeds is None:
controlnet_pooled_projections = torch.zeros_like(pooled_prompt_embeds)
elif controlnet_pooled_projections is None:
controlnet_pooled_projections = pooled_prompt_embeds
```
Given that in training, the pooled_prompt_embeds are fed to the model:
https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/examples/controlnet/train_controlnet_sd3.py#L1293
Additionally, I am wondering if this line:
https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/examples/controlnet/train_controlnet_sd3.py#L1287
Should be aligned with this line:
https://github.com/huggingface/diffusers/blob/a3e8d3f7deed140f57a28d82dd0b5d965bd0fb09/examples/controlnet/train_controlnet_sd3.py#L1257
This seems to be the more sensible approach, but will probably not make much difference since the ControlNet can also learn the shift. It might speed up convergence *slightly*.
Best,
Tobias
### Reproduction
Train an SD3 ControlNet and during log_validation it will be executed.
### Logs
_No response_
### System Info
diffusers==0.30.3
### Who can help?
@yiyixuxu @sayakpaul
Guía de contribución
Línea de trabajo
Comienza con el bloque if-else alrededor de la línea 977 en src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet.py y, después, compáralo con el manejo de pooled prompt alrededor de las líneas 1287 y 1293 en examples/controlnet/train_controlnet_sd3.py. Ejecuta la ruta log_validation indicada después del entrenamiento y verifica que el entrenamiento y la inferencia utilicen pooled projections alineadas, incluida una decisión resuelta sobre el shift en la línea 1257.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python, pytorch
- Área
- machine-learning
- Tipo de issue
- Error
- Dificultad
- 3/5
- Tiempo estimado
- 1-2 días
- Estado de actividad
- Estancado
- Claridad
- Bastante claro
- Aptitud para principiantes
- 45/100