Conv3D Backpropagation gets slower in mixprecision

Hi,

After converting my program into mixed-precision using amp, the forward time gets shorter while the backward time gets longer when I record running time by “import time”.

Then I use “torch.profiler” to record the running time.
However, it seems that the result of torch.profiler meets my expectation: cuda.time becomes around 1/4 of the fp32.

with torch.autograd.profiler.profile(use_cuda=True) as prof:
        for i, (input, target) in enumerate(train_loader):     
            input = input.cuda(non_blocking=True)
            target = target.cuda(non_blocking=True)  
                
            torch.cuda.synchronize()
            time0 = time.time()
            
            output = model(input)
            loss = criterion(output, target)     
            
            torch.cuda.synchronize()
            time1 = time.time()
            
            optimizer.zero_grad()        
            #loss.sum().backward()
            with amp.scale_loss(loss, optimizer) as scaled_loss:
                scaled_loss.sum().backward()
                
            torch.cuda.synchronize()
            time2 = time.time()            
            
            optimizer.step()
            
            torch.cuda.synchronize()
            time3 = time.time()

            batch_time.update(time3 - end)
            data_time.update(time0 - end)
            forward_time.update(time1-time0)
            backward_time.update(time2-time1)
            comm_time.update(time3-time2)
                
            torch.cuda.synchronize()        
            end = time.time()
            
            if i % print_freq == 0:
                print('Rank: [{rank}]\t'
                'Train: [{epoch}][{batch}/{batchs}]\t'
                'SUM {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                'DT {data_time.val:.3f} ({data_time.avg:.3f})\t'
                'FW {forward_time.val:.3f} ({forward_time.avg:.3f})\t'
                'BK {backward_time.val:.3f} ({backward_time.avg:.3f})\t'
                'CM {comm_time.val:.3f} ({comm_time.avg:.3f})\t'
                .format(rank=torch.distributed.get_rank(), epoch=epoch, batch=i, batchs=len(train_loader), batch_time=batch_time,
                data_time=data_time, forward_time=forward_time, backward_time=backward_time, comm_time=comm_time)) 
    print(prof.key_averages().table(sort_by="self_cpu_time_total"))

mixed-precision:
import time :
Entire Epoch: [1] Train: [0] SUM: 1828.379 DT: 13.270 FW: 71.987 BK: 1711.248 CM: 31.873
torch profiler:
Self CPU time total: 56.883s
CUDA time total: 893.112s

FP32:
import time :
Entire Epoch: [3] Train: [0] SUM: 1584.260 DT: 15.714 FW: 368.550 BK: 1119.231 CM: 80.766
torch profiler:
Self CPU time total: 105.049s
CUDA time total: 3021.355s

Here my question is: Why here two recorders give different results?
My model uses conv3d and I run my code on Tesla V100.
The input channel and output channel of our model is divisible by 8.
Since the model is large, here the batch size is only 1.

Detailed Logs of my program are listed below:
http://49.234.107.127:81/index.php/s/qa3Yjo8WJwNZjCS (mixed precision)
http://49.234.107.127:81/index.php/s/y8SpyfiM3d5SZp7

Many thanks.

Start with the inline timing code around model(input), scaled_loss.sum().backward(), optimizer.step(), and torch.autograd.profiler.profile. Reproduce the mixed-precision and FP32 measurements on the Tesla V100, compare the host and CUDA timings, and determine what explains their discrepancy; done means a documented explanation supported by the profiling results.