[Bug] Gradients not synchronized

https://github.com/kohya-ss/sd-scripts/blob/2a23713f71628b2d1b88a51035b3e4ee2b5dbe46/fine_tune.py#L247

I have not idea what this line is used for, but this unwrap DDP module so that the training process become unsynchronized, i.e, no gradient communication in multi-gpu training, each node trains independently with part of the data.

I ensure this by adding sleep in one of the worker and finding no hang in the main training process. And by deleting this line the job got properly synchronized.

Start at fine_tune.py around line 247 and inspect how the DDP module is unwrapped during training. Reproduce the reported multi-GPU case, including a worker delay, and verify that gradients are communicated and the training process remains synchronized without removing required behavior.

Written by the indexing model from the issue text.