Should I reduce the loss before the backward function?

Recently, I do some experiments with the application of semantic segmentation.
There is over 1 point performance gap between two experiments:
1.

dist.all_reduce(loss, dist.ReduceOp.SUM)
loss.div_(world_size)

optimizer.zero_grad()
loss.backward()
optimizer.step()

Here, the loss applied backward function is the reduced loss.

# the reduced loss is just used for log
reduced_loss = loss.clone().detach()
dist.all_reduce(reduced_loss , dist.ReduceOp.SUM)
reduced_loss .div_(world_size)

optimizer.zero_grad()
loss.backward()
optimizer.step()

Here, the loss applied backward function is the original loss, while the reduced loss is simply used for logging.

Two experiments have the same setting excluding the loss reduction mentioned above.

According to my understanding of distributed training, the manner 2 is the correct one.
However, the manner 1 can achieve higher performance than manner 2.

Therefore, I want to know which one is correct? Or what is the correct method to handle the learning rate, batch size and loss?

No source files, tests, or entry points are mentioned. Start by reviewing the distributed-training guidance relevant to the two loss-reduction snippets, then document which approach is correct and how loss reduction relates to learning rate and batch size; done means the question has an explicit, reproducible explanation.