huggingface / huggingface/diffusers

training example for instruct pix2pix doesn't zero out embeds

Abierto
#7,920 9 comentarios 0 reacciones 0 asignados Ver en GitHub
bug stale
Lenguaje dominante
Python
Estrellas
34.5k
Forks
7.3k
Merge medio
3 d 3 h
PR fusionados (30 d)
91

Descripción

### Describe the bug

When running inference on SDXL, the config specifies to zero out the embedding when the prompt is empty.

### Reproduction

```py
# Get null conditioning
def compute_null_conditioning():
null_conditioning_list = []
for a_tokenizer, a_text_encoder in zip(tokenizers, text_encoders):
null_conditioning_list.append(
a_text_encoder(
tokenize_captions([""], tokenizer=a_tokenizer).to(accelerator.device),
output_hidden_states=True,
).hidden_states[-2]
)
return torch.concat(null_conditioning_list, dim=-1)

null_conditioning = compute_null_conditioning()
```

this could likely be replaced with a probabilistic call to `torch.zeros_like()` inside the training loop instead.

I've checked the values of the embeds, and classifier-free guidance at inference time definitely makes use of the zero embed and not just `""`, which end up producing very different results.

other models though like deepfloyd just use `""` from eg. T5 and behave rather differently.

### Logs

_No response_

### System Info

N/A

### Who can help?

@sayakpaul

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Start in the instruct pix2pix training example, inspect compute_null_conditioning and the training loop, and compare their conditioning behavior with the SDXL inference configuration. Done means the example handles empty-prompt conditioning consistently with the configured zero embeddings without changing the behavior required by other model families.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python, pytorch
Área
machine-learning
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.