When to validate and save model while using apex's DDP

When I use SyncBN and apex.parallel.DistributedDataParallel together, I found that all print() functions, both in train and validate part, print twice. The problem is, I save best model by validating periodly, and I found that there are two best models saved by two GPUs. I don't know what to do.

Then, I checked this file, and found this line
if batch_idx % args.log_interval == 0 and args.local_rank == 0:
and this
if args.local_rank == 0:.

So you only print training loss and do validation when args.local_rank == 0, which I think equals to gpu=0 in my case. Does this means gpu 0 is the master and gpu 1 is slave, only gpu 0 would compute the total loss, just like what torch.nn.DataParallel do?

And If I want to do validation and save best model, do I only need to validate on gpu 0 and only save the model on gpu0? Then if I want to use saved model, just like dealing with the model saved when using torch.nn.DataParallel?

Sorry for a rookie's question.Thanks

Start with examples/distributed/main.py and the uses of args.local_rank shown in the issue. Clarify the expected validation, logging, and checkpoint-saving behavior when using SyncBN with apex.parallel.DistributedDataParallel, including how a saved model should be reused.