How to make the RegionProposalNetwork generate more proposals in FasterRCNN?

I'm trying to update the proposal losses function of MaskRCNN to increase the recall. I'm trying to do this by adding a positive weight to the BCE function

How I create my proposal losses function:

CLASS_WEIGHTS = torch.tensor([50])

def compute_loss(
        objectness: Tensor, pred_bbox_deltas: Tensor, labels: List[Tensor], regression_targets: List[Tensor]
    ) -> Tuple[Tensor, Tensor]:
    """
    Args:
        objectness (Tensor)
        pred_bbox_deltas (Tensor)
        labels (List[Tensor])
        regression_targets (List[Tensor])

    Returns:
        objectness_loss (Tensor)
        box_loss (Tensor)
    """

    sampled_pos_inds, sampled_neg_inds = model.rpn.fg_bg_sampler(labels)
    sampled_pos_inds = torch.where(torch.cat(sampled_pos_inds, dim=0))[0]
    sampled_neg_inds = torch.where(torch.cat(sampled_neg_inds, dim=0))[0]

    sampled_inds = torch.cat([sampled_pos_inds, sampled_neg_inds], dim=0)

    objectness = objectness.flatten()

    labels = torch.cat(labels, dim=0)
    regression_targets = torch.cat(regression_targets, dim=0)

    box_loss = F.smooth_l1_loss(
        pred_bbox_deltas[sampled_pos_inds],
        regression_targets[sampled_pos_inds],
        beta=1 / 9,
        reduction="sum",
    ) / (sampled_inds.numel())

    objectness_loss = F.binary_cross_entropy_with_logits(objectness[sampled_inds], labels[sampled_inds],
                                                        pos_weight=CLASS_WEIGHTS # USE CLASS WEIGHT HERE
                                                        )
    return objectness_loss, box_loss

Then how I set the model to use this proposal losses function:

model = maskrcnn_resnet50_fpn(weights=MaskRCNN_ResNet50_FPN_Weights.DEFAULT)
model.rpn.compute_loss = compute_loss

When I train the model now:

• the loss increases significantly (e.g. before it was 1, now it is like 50, which is expected)
• BUT the recall stays around the same (e.g. stagnates around 0.55 after training for several epochs)

Why is this the case? How do I get the recall to improve (i.e. how do I generate more proposals)?

FYI: I already tried setting the score threshold to 0, this didn't do anything either…

Start by reading the model.rpn.compute_loss entry point used with maskrcnn_resnet50_fpn, then trace how binary_cross_entropy_with_logits and the score threshold affect proposal selection. Reproduce the reported loss and recall measurements while checking the proposal-generation settings; done means the cause is explained and a documented change produces a measurable recall improvement.