Changing instance and category values in Point Rend has no effect on point sampling stages.

## Instructions To Reproduce the 🐛 Bug:
1. Full runnable code or full changes you made:
```
# You may need to restart your runtime prior to this, to let your installation take effect
# Some basic setup:
# Setup detectron2 logger
import detectron2
from detectron2.utils.logger import setup_logger
setup_logger()

# import some common libraries
import numpy as np
import cv2
import torch

# import some common detectron2 utilities
from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer, ColorMode
from detectron2.data import MetadataCatalog
coco_metadata = MetadataCatalog.get("coco_2017_val")

# import PointRend project
from detectron2.projects import point_rend

im = cv2.imread("/homelocal/images/chair_table.jpg")
im = cv2.resize(im, (0, 0), fx=0.5, fy=0.5)

cfg = get_cfg()
# Add PointRend-specific config
point_rend.add_pointrend_config(cfg)
# Load a config from file
cfg.merge_from_file("projects/PointRend/configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set threshold for this model
# Use a model from PointRend model zoo: https://github.com/facebookresearch/detectron2/tree/master/projects/PointRend#pretrained-models
cfg.MODEL.WEIGHTS = "detectron2://PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco/164254221/model_final_736f5a.pkl"
predictor = DefaultPredictor(cfg)
outputs = predictor(im)

# First we define a simple function to help us plot the intermediate representations.
import matplotlib.pyplot as plt

def plot_mask(mask, title="", point_coords=None, figsize=10, point_marker_size=5):
'''
Simple plotting tool to show intermediate mask predictions and points
where PointRend is applied.

Args:
mask (Tensor): mask prediction of shape HxW
title (str): title for the plot
point_coords ((Tensor, Tensor)): x and y point coordinates
figsize (int): size of the figure to plot
point_marker_size (int): marker size for points
'''

H, W = mask.shape
plt.figure(figsize=(figsize, figsize))
if title:
title += ", "
plt.title("{}resolution {}x{}".format(title, H, W), fontsize=30)
plt.ylabel(H, fontsize=30)
plt.xlabel(W, fontsize=30)
plt.xticks([], [])
plt.yticks([], [])
plt.imshow(mask, interpolation="nearest", cmap=plt.get_cmap('turbo'))
if point_coords is not None:
plt.scatter(x=point_coords[0], y=point_coords[1], color="red", s=point_marker_size, clip_on=True)
plt.xlim(-0.5, W - 0.5)
plt.ylim(H - 0.5, - 0.5)
plt.show()

from detectron2.data import transforms as T
model = predictor.model
# In this image we detect several objects but show only the first one.
instance_idx = 0
# Mask predictions are class-specific, "plane" class has id 4.
category_idx = 62

with torch.no_grad():
# Prepare input image.
height, width = im.shape[:2]
im_transformed = T.ResizeShortestEdge(800, 1333).get_transform(im).apply_image(im)
batched_inputs = [{"image": torch.as_tensor(im_transformed).permute(2, 0, 1)}]

# Get bounding box predictions first to simplify the code.
detected_instances = [x["instances"] for x in model.inference(batched_inputs)]
[r.remove("pred_masks") for r in detected_instances] # remove existing mask predictions
pred_boxes = [x.pred_boxes for x in detected_instances]

# Run backbone.
images = model.preprocess_image(batched_inputs)
features = model.backbone(images.tensor)

# Given the bounding boxes, run coarse mask prediction head.
mask_coarse_logits = model.roi_heads.mask_head.coarse_head(model.roi_heads.mask_head._roi_pooler(features, pred_boxes))

plot_mask(
mask_coarse_logits[instance_idx, category_idx].to("cpu"),
title="Coarse prediction"
)

# Prepare features maps to use later
mask_features_list = [
features[k] for k in model.roi_heads.mask_head.mask_point_in_features
]
features_scales = [
model.roi_heads.mask_head._feature_scales[k]
for k in model.roi_heads.mask_head.mask_point_in_features
]

from detectron2.layers import interpolate
from detectron2.projects.point_rend.mask_head import calculate_uncertainty
from detectron2.projects.point_rend.point_features import (
get_uncertain_point_coords_on_grid,
point_sample,
point_sample_fine_grained_features,
)

num_subdivision_steps = 7
num_subdivision_points = 24 * 24

with torch.no_grad():
# We take predicted classes, whereas during real training ground truth classes are used.
pred_classes = torch.cat([x.pred_classes for x in detected_instances])
plot_mask(
mask_coarse_logits[0, category_idx].to("cpu").numpy(),
title="Coarse prediction"
)

mask_logits = mask_coarse_logits
for subdivions_step in range(num_subdivision_steps):
# Upsample mask prediction
mask_logits = interpolate(
mask_logits, scale_factor=2, mode="bilinear", align_corners=False
)
# If `num_subdivision_points` is larger or equalto to the
# resolution of the next step, then we can skip this step
H, W = mask_logits.shape[-2:]
if (
num_subdivision_points >= 4 * H * W
and subdivions_step < num_subdivision_steps - 1
):
continue
# Calculate uncertainty for all points on the upsampled regular grid
uncertainty_map = calculate_uncertainty(mask_logits, pred_classes)
# Select most `num_subdivision_points` uncertain points
point_indices, point_coords = get_uncertain_point_coords_on_grid(
uncertainty_map,
num_subdivision_points
)

# Extract fine-grained and coarse features for the points
fine_grained_features, _ = point_sample_fine_grained_features(
mask_features_list, features_scales, pred_boxes, point_coords
)
coarse_features = point_sample(mask_coarse_logits, point_coords, align_corners=False)

# Run PointRend head for these points
point_logits = model.roi_heads.mask_head.point_head(fine_grained_features, coarse_features)

# put mask point predictions to the right places on the upsampled grid.
R, C, H, W = mask_logits.shape
x = (point_indices[instance_idx] % W).to("cpu")
y = (point_indices[instance_idx] // W).to("cpu")
point_indices = point_indices.unsqueeze(1).expand(-1, C, -1)
mask_logits = (
mask_logits.reshape(R, C, H * W)
.scatter_(2, point_indices, point_logits)
.view(R, C, H, W)
)
plot_mask(
mask_logits[instance_idx, category_idx].to("cpu"),
title="Subdivision step: {}".format(subdivions_step + 1),
point_coords=(x, y)
)

# This code has been copied from https://colab.research.google.com/drive/1isGPL5h5_cKoPPhVL9XhMokRtHDvmMVL#scrollTo=CIny-6sotDCL
```
2. What exact command you run:
```
python intermediate.py
```
3. __Full logs__ or other relevant observations:
```
The output shows different level of coarse and fine images. However, the code only selects a particular chair from the image. I have appended all the output images as well as the instance segmentation into one single image here to show the results

```

4. please simplify the steps as much as possible so they do not require additional resources to
run, such as a private dataset.

## Expected behavior:

Changing the instance_idx and category_idx to relevant values does not affect the code. For example in this case, changing category_idx=62 (chair), to categroy_idx=4(plane) still yeild same results.

Also changing the instance from 0 to 1 should allow for capturing any other chair but it doesnt.

And I would like to know where do we increase/ decrease the size of crop such that a larger area is captured for coarse and fine operations rather than just a single chair?

## Environment:

Provide your environment information using the following command:
This command gives me error 404
```
wget -nc -q https://github.com/facebookresearch/detectron2/raw/master/detectron2/utils/collect_env.py && python collect_env.py
```
Nonetheless, the env details are as follows
```

sys.platform linux
Python 3.8.10 (default, Jun 4 2021, 15:09:15) [GCC 7.5.0]
numpy 1.20.1
detectron2 0.6 @/homelocal/detectron2/detectron2
Compiler GCC 9.3
CUDA compiler CUDA 11.3
detectron2 arch flags 5.2, 6.0, 6.1, 7.0, 7.5, 8.0, 8.6
DETECTRON2_ENV_MODULE
PyTorch 1.9.0a0+2ecb2c7 @/opt/conda/lib/python3.8/site-packages/torch
PyTorch debug build False
GPU available Yes
GPU 0,1,2,3 Tesla V100-DGXS-32GB (arch=7.0)
Driver version 450.80.02
CUDA_HOME /usr/local/cuda
TORCH_CUDA_ARCH_LIST 5.2 6.0 6.1 7.0 7.5 8.0 8.6+PTX
Pillow 8.2.0
torchvision 0.9.0a0 @/opt/conda/lib/python3.8/site-packages/torchvision
torchvision arch flags 5.2, 6.0, 6.1, 7.0, 7.5, 8.0, 8.6
fvcore 0.1.5.post20211023
iopath 0.1.9
cv2 3.4.11

___________________________________________________________________________________________________

```