huggingface / huggingface/diffusers

[SD3 DreamBooth] Cached latents can be paired with the wrong per-image captions when shuffle=True

Abierto
#14,430 1 comentario 0 reacciones 0 asignados Ver en GitHub
bug
Lenguaje dominante
Python
Estrellas
34.5k
Forks
7.3k
Merge medio
3 d 3 h
PR fusionados (30 d)
91

Descripción

### Describe the bug

The SD3 DreamBooth LoRA trainer creates the training dataloader with shuffle enabled.
When latent caching is enabled, the script traverses the shuffled dataloader once and stores the VAE latent distributions in a list. Training then starts another traversal of the same shuffled dataloader, but retrieves cached latents using only the current batch position.
Because the second traversal can produce a different sample order, a per-image caption in the current batch may be paired with the cached latent of another image. This problem is mostly hidden when every image uses the same instance prompt, but it can silently corrupt training when the caption column contains a different caption for each image.
Cached latents should be stored and retrieved using a stable dataset index or sample ID instead of the dataloader step. This would preserve shuffling while keeping each image, latent and caption correctly associated.
A similar correspondence issue was reported for the Qwen-Image trainer in issue #12124.

### Reproduction

The issue follows directly from the current data flow in train_dreambooth_lora_sd3.py.
The training dataloader is created with shuffle enabled. When latent caching is enabled, the first traversal appends each latent distribution to a list. Training starts a second traversal of the shuffled dataloader and reads the cached latent using latents_cache[step].
The first and second traversals are not guaranteed to have the same sample order. Therefore, the current per-image caption and latents_cache[step] are not guaranteed to belong to the same dataset sample.
This can be observed by printing the sample index during the latent caching traversal and again during the first training epoch.

### Logs

```shell
No runtime exception is raised. This is a silent image-caption correspondence issue that can affect the learned result without producing an error message.
```

### System Info

Diffusers: 0.40.0.dev0
Diffusers source commit: d6726f38a0c5ca6c06a8f227fb7bade3486ed98d
Python: 3.12
PyTorch: 2.8.0
CUDA: 12.8
Platform: Ubuntu 22.04
GPU: NVIDIA GeForce RTX 4090, 24GB
Training script: examples/dreambooth/train_dreambooth_lora_sd3.py
Model: stabilityai/stable-diffusion-3.5-medium
Dataset mode: local ImageFolder with per-image captions
Latent caching: enabled
Dataloader shuffling: enabled

### Who can help?

@sayakpaul @DN6

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza en examples/dreambooth/train_dreambooth_lora_sd3.py y sigue el recorrido del almacenamiento en caché de latents junto con la primera época de entrenamiento, centrándote en cómo se asocia latents_cache[step] con cada muestra. Reproduce el problema con datos de ImageFolder barajados y captions distintas para cada imagen; se considera terminado cuando los latents almacenados en caché siguen asociados a sus muestras y captions originales durante ambos recorridos.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python, pytorch
Área
machine-learning
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Tranquilo
Claridad
Bien especificado
Aptitud para principiantes
68/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.