FCOS empty box images

I am playing around with new FCOS models (thanks for that) and am encountering issues when providing images without box annotations. This is a common use case in object detection, and also works for other detector models in torchvision.

A simple example to replicate:

model = fcos_resnet50_fpn(pretrained=True)
model(torch.zeros((1,3,512,512)), targets=[{"boxes": torch.empty(0,4), "labels": torch.empty(0,1).to(torch.int64)}])

An indexing error happens in FCOSHead when running compute_loss in this part:

all_gt_classes_targets = []
all_gt_boxes_targets = []
for targets_per_image, matched_idxs_per_image in zip(targets, matched_idxs):
    gt_classes_targets = targets_per_image["labels"][matched_idxs_per_image.clip(min=0)]
    gt_classes_targets[matched_idxs_per_image < 0] = -1  # backgroud
    gt_boxes_targets = targets_per_image["boxes"][matched_idxs_per_image.clip(min=0)]
    all_gt_classes_targets.append(gt_classes_targets)
    all_gt_boxes_targets.append(gt_boxes_targets)

A workaround seems to be necessary, when having empty targets. Happy for any guidance, maybe there is also a different way necessary for me to train on empty images.

@jdsgomes @xiaohu2015 @zhiqwang

Versions

Torchvision @ master

Start with the FCOSHead compute_loss code shown in the issue and run the provided fcos_resnet50_fpn reproduction using an image with empty boxes and labels. Trace the indexing failure for empty targets; done means FCOS accepts such inputs without an indexing error and preserves the expected loss behavior for empty images.