huggingface / huggingface/diffusers
[Training] Resume checkpoint global step inconsistent/confusion across scripts
- Lenguaje dominante
- Python
- Estrellas
- 34.5k
- Forks
- 7.3k
- Merge medio
- 3 d 3 h
- PR fusionados (30 d)
- 91
Descripción
### Describe the bug
Hi,
I have been working on training scripts for multiple models (T2I, IP2P) and found the different logic to calculate `step` and `epoch` while resuming training different across scripts.
In `train_text_to_image.py` script [link](https://github.com/huggingface/diffusers/blob/fe5f035f797a5fa663a98030c9d0ec2f982cd09d/examples/text_to_image/train_text_to_image.py#L902)
```
accelerator.print(f"Resuming from checkpoint {path}")
accelerator.load_state(os.path.join(args.output_dir, path))
global_step = int(path.split("-")[1])
initial_global_step = global_step
first_epoch = global_step // num_update_steps_per_epoch
```
In `train_instruct_pix2pix.py` script [link](https://github.com/huggingface/diffusers/blob/fe5f035f797a5fa663a98030c9d0ec2f982cd09d/examples/instruct_pix2pix/train_instruct_pix2pix.py#L825)
```
accelerator.print(f"Resuming from checkpoint {path}")
accelerator.load_state(os.path.join(args.output_dir, path))
global_step = int(path.split("-")[1])
resume_global_step = global_step * args.gradient_accumulation_steps
first_epoch = global_step // num_update_steps_per_epoch
resume_step = resume_global_step % (num_update_steps_per_epoch * args.gradient_accumulation_steps)
```
In the similar [issue](https://github.com/huggingface/diffusers/issues/5005), some changes are made for the progress bar inconsistency but I am bit confused with the following things:-
1. The multiplication of `args.gradient_accumulation_steps` in `train_instruct_pix2pix.py` script
2. In general, when does global-step indicate and how does it's being updated, in both the scripts I can see the following code but couldn't understand it from `accelerate` documentation
```
if accelerator.sync_gradients:
if args.use_ema:
ema_unet.step(unet.parameters())
progress_bar.update(1)
global_step += 1
accelerator.log({"train_loss": train_loss}, step=global_step)
train_loss = 0.0
```
If we are using multiple GPUs with gradient accumulation, at what event `global_step` is updated- is it being updated independently by each GPU (since the code is not wrapped with `accelerator.is_main_process`), also how accumulation affecting the tracking here?
### Reproduction
-
### Logs
_No response_
### System Info
-
### Who can help?
@sayakpaul
Guía de contribución
Línea de trabajo
Comienza comparando la lógica de reanudación en examples/text_to_image/train_text_to_image.py y examples/instruct_pix2pix/train_instruct_pix2pix.py; después, lee el bloque accelerator.sync_gradients que actualiza global_step. Analiza cómo afectan la acumulación de gradientes y los múltiples procesos a estos valores, y define un comportamiento coherente de reanudación y seguimiento del progreso en ambos scripts.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python, pytorch
- Área
- machine-learning
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 28/100