facebookresearch / facebookresearch/detectron2

numpy() fails when visualizing an output element of a input batch during inference

Open
#3,102 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

Hi there,
i tried to visualize the outputs after a batch inference.

```
### On my PC I am using detectron2 v0.4, commit-ID: 13afb035142734a309b20634dadbba0504d7eefe
### but I was able to reproduce the error with your colab (https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5)

### cell 5 modified to:
cfg = get_cfg()
# add project-specific config (e.g., TensorMask) here if you're not running a model in detectron2's core library
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set threshold for this model
# Find a model from detectron2's model zoo. You can use the https://dl.fbaipublicfiles... url as well
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg)
import PIL
img_tensor = torchvision.transforms.PILToTensor()(PIL.Image.open("./input.jpg"))
img_batch = [{"image": img_tensor} for i in range(2)]
outputs = predictor.model(img_batch)

### cell 6 modified to (here the bug reveals itself, since scores and pred_boxes are still attached):
# look at the outputs. See https://detectron2.readthedocs.io/tutorials/models.html#model-output-format for specification
print(outputs[0]["instances"].pred_classes)
print(outputs[0]["instances"].pred_boxes)
print(outputs[0]["instances"].scores)

### cell 7 modified to:
# We can use `Visualizer` to draw the predictions on the image.
v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
out = v.draw_instance_predictions(outputs[0]["instances"].to("cpu"))
cv2_imshow(out.get_image()[:, :, ::-1])
```

Since some of the output objects, have tensors that are still using gradients, I get the following error when trying to visualize the results (after running cell 7):
```
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
in ()
1 # We can use `Visualizer` to draw the predictions on the image.
2 v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
----> 3 out = v.draw_instance_predictions(outputs[0]["instances"].to("cpu"))
4 cv2_imshow(out.get_image()[:, :, ::-1])

2 frames
/usr/local/lib/python3.7/dist-packages/detectron2/utils/visualizer.py in _convert_boxes(self, boxes)
1178 """
1179 if isinstance(boxes, Boxes) or isinstance(boxes, RotatedBoxes):
-> 1180 return boxes.tensor.numpy()
1181 else:
1182 return np.asarray(boxes)

RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
```

## Workaround (run this before visualizing the results):
```
outputs[0]["instances"] = outputs[0]["instances"].to("cpu")
outputs[0]["instances"].pred_boxes.tensor = outputs[0]["instances"].pred_boxes.tensor.detach()
outputs[0]["instances"].scores = outputs[0]["instances"].scores.detach()
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the modified Colab cells and inspect detectron2/utils/visualizer.py at _convert_boxes, especially the boxes.tensor.numpy() call shown in the traceback. Verify visualization of gradient-attached inference outputs without the reported workaround, and confirm the existing workaround behavior remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
computer-vision
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.