how is multi gpu loss gathered?

I've been looking into the sd3 train branch, im trying to understand how are the loss gathered for multi-gpu and would love to understand the logic behind it.
I'm used to working with accelerator.gather/reduce for loss/tensor updates. however im not seeing any of that being used in the sd3 training script which got me curious - how are the losses gathered across all processes

Start with the SD3 training script and trace how the loss is computed and updated across processes. Compare that flow with the mentioned accelerator.gather/reduce operations. Done means documenting whether losses are gathered across all processes and explaining the logic used instead.

Written by the indexing model from the issue text.