I am not able to obtain results with custom backbone

_I am following the tutorial about FasterRCNN and I would like to test my network as backbone of the net:

UCapsNet return 512 features maps
I am training on VocPascal 2007_

FRCN_model = FasterRCNN(backbone_model.Ucapsnet, 21, rpn_anchor_generator=backbone_model.anchor_generator, box_roi_pool=backbone_model.roi_pooler)
FRCN_model = FRCN_model.to(device)

params = [p for p in FRCN_model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.02, momentum=0.9, weight_decay=1e-4)

pbar = tqdm(range(n_epochs))
for epoch in pbar:
train_one_epoch(FRCN_model, optimizer, dataloaders['train'], device, epoch, print_freq=10)
evaluate(FRCN_model, dataloaders['val'], device=device)

I got:
Averaged stats: model_time: 1605886336.0000 (1605886304.8101) evaluator_time: 0.0275 (0.0285)
Accumulating evaluation results...
DONE (t=0.06s).
IoU metric: bbox
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

In training, the loss is dropping slowly to 1.15 but in evaluation, i do not get anything.

Please help me understand

cc @fmassa

Begin with the FasterRCNN tutorial and the custom UCapsNet, anchor_generator, and roi_pooler setup; compare the backbone's expected outputs with what the training and evaluation calls receive. Reproduce the reported VOC Pascal 2007 run and inspect the zero AP/AR output. Done means identifying a concrete project change or configuration that makes evaluation produce detections.