:If you _really_ know what you are doing, you can disable this warning by passing allow_banned=True to `amp.init()`.

enabled : True
[1,1]:loss_scale : dynamic
[1,1]:master_weights : None
[1,1]:patch_torch_functions : True
[1,1]:opt_level : O1
[1,1]:keep_batchnorm_fp32 : None
[1,1]:cast_model_type : None
[1,1]:Processing user overrides (additional kwargs that are not None)...
[1,1]:After processing overrides, optimization options are:
[1,1]:enabled : True
[1,1]:loss_scale : dynamic
[1,1]:master_weights : None
[1,1]:patch_torch_functions : True
[1,1]:opt_level : O1
[1,1]:keep_batchnorm_fp32 : None
[1,1]:cast_model_type : None
[1,0]:
[1,0]:
[1,0]:epoch: 0,batch: 0[1,0]:
[1,0]:Traceback (most recent call last):
[1,0]: File "train_1.py", line 187, in
[1,0]: loss, outputs = model(imgs, targets)
[1,0]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 547, in call
[1,0]: result = self.forward(*input, **kwargs)
[1,0]: File "/media/ai/sdc1/tools/horovod_study/yolov3-pytoch/PyTorch-YOLOv3/models.py", line 260, in forward
[1,0]: x, layer_loss = module[0](x, targets, img_dim)
[1,0]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 547, in call
[1,0]: result = self.forward(*input, **kwargs)
[1,0]: File "/media/ai/sdc1/tools/horovod_study/yolov3-pytoch/PyTorch-YOLOv3/models.py", line 197, in forward
[1,0]: loss_conf_obj = self.bce_loss(pred_conf[obj_mask], tconf[obj_mask])
[1,0]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 547, in call
[1,0]: result = self.forward(*input, **kwargs)
[1,0]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py", line 498, in forward
[1,0]: return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
[1,0]: File "/usr/local/lib/python3.5/dist-packages/apex/amp/wrap.py", line 124, in wrapper
[1,0]: raise NotImplementedError(custom_err_msg)
[1,0]:NotImplementedError:
[1,0]:amp does not work out-of-the-box with F.binary_cross_entropy or torch.nn.BCELoss. It requires that the output of the previous function be already a FloatTensor.
[1,0]:
[1,0]:Most models have a Sigmoid right before BCELoss. In that case, you can use
[1,0]: torch.nn.BCEWithLogitsLoss
[1,0]:to combine Sigmoid+BCELoss into a single layer that is compatible with amp.
[1,0]:Another option is to add
[1,0]: amp.register_float_function(torch, 'sigmoid')
[1,0]:before calling amp.init().
[1,0]:If you really know what you are doing, you can disable this warning by passing allow_banned=True to amp.init().
[1,1]:Traceback (most recent call last):
[1,1]: File "train_1.py", line 187, in
[1,1]: loss, outputs = model(imgs, targets)
[1,1]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 547, in call
[1,1]: result = self.forward(*input, **kwargs)
[1,1]: File "/media/ai/sdc1/tools/horovod_study/yolov3-pytoch/PyTorch-YOLOv3/models.py", line 260, in forward
[1,1]: x, layer_loss = module[0](x, targets, img_dim)
[1,1]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 547, in call
[1,1]: result = self.forward(*input, **kwargs)
[1,1]: File "/media/ai/sdc1/tools/horovod_study/yolov3-pytoch/PyTorch-YOLOv3/models.py", line 197, in forward
[1,1]: loss_conf_obj = self.bce_loss(pred_conf[obj_mask], tconf[obj_mask])
[1,1]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 547, in call
[1,1]: result = self.forward(*input, **kwargs)
[1,1]: File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py", line 498, in forward
[1,1]: return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
[1,1]: File "/usr/local/lib/python3.5/dist-packages/apex/amp/wrap.py", line 124, in wrapper
[1,1]: raise NotImplementedError(custom_err_msg)
[1,1]:NotImplementedError:
[1,1]:amp does not work out-of-the-box with F.binary_cross_entropy or torch.nn.BCELoss. It requires that the output of the previous function be already a FloatTensor.
[1,1]:
[1,1]:Most models have a Sigmoid right before BCELoss. In that case, you can use
[1,1]: torch.nn.BCEWithLogitsLoss
[1,1]:to combine Sigmoid+BCELoss into a single layer that is compatible with amp.
[1,1]:Another option is to add
[1,1]: amp.register_float_function(torch, 'sigmoid')
[1,1]:before calling amp.init().
[1,1]:If you really know what you are doing, you can disable this warning by passing allow_banned=True to amp.init().

Start with the reported call in train_1.py at line 187 and the loss calculation in models.py at line 197, then read the Apex wrapper at apex/amp/wrap.py line 124. Compare the failing BCELoss path with the amp.init configuration shown in the traceback; the issue is done when the compatibility problem and a reproducible resolution are clearly established.