How to use a single-machine multi-GPU to run

The function to get the device: init_distributed_device(args), in training.main.py line 129, it seems that only a single-GPU device can be obtained. The key part of the function is defined as follows:

  if torch.cuda.is_available():  
        if args.distributed and not args.no_set_device_rank: 
            device = 'cuda:%d' % args.local_rank 
        else:  
            device = 'cuda:0'  
        torch.cuda.set_device(device)  
    else:
        device = 'cpu'

How to make the project run on a single-machine multi-GPU?

Start in training.main.py around line 129 and inspect init_distributed_device(args), especially the args.distributed, args.no_set_device_rank, and args.local_rank branches. Trace how training is launched and configured, then establish the documented single-machine multi-GPU invocation and verify that training uses multiple GPUs without selecting only cuda:0.