facebookresearch / facebookresearch/detectron2
numpy() fails when visualizing an output element of a input batch during inference
- Dominant language
- Python
- Stars
- 34.7k
- Forks
- 7.9k
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
i tried to visualize the outputs after a batch inference.
```
### On my PC I am using detectron2 v0.4, commit-ID: 13afb035142734a309b20634dadbba0504d7eefe
### but I was able to reproduce the error with your colab (https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5)
### cell 5 modified to:
cfg = get_cfg()
# add project-specific config (e.g., TensorMask) here if you're not running a model in detectron2's core library
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 # set threshold for this model
# Find a model from detectron2's model zoo. You can use the https://dl.fbaipublicfiles... url as well
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
predictor = DefaultPredictor(cfg)
import PIL
img_tensor = torchvision.transforms.PILToTensor()(PIL.Image.open("./input.jpg"))
img_batch = [{"image": img_tensor} for i in range(2)]
outputs = predictor.model(img_batch)
### cell 6 modified to (here the bug reveals itself, since scores and pred_boxes are still attached):
# look at the outputs. See https://detectron2.readthedocs.io/tutorials/models.html#model-output-format for specification
print(outputs[0]["instances"].pred_classes)
print(outputs[0]["instances"].pred_boxes)
print(outputs[0]["instances"].scores)
### cell 7 modified to:
# We can use `Visualizer` to draw the predictions on the image.
v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
out = v.draw_instance_predictions(outputs[0]["instances"].to("cpu"))
cv2_imshow(out.get_image()[:, :, ::-1])
```
Since some of the output objects, have tensors that are still using gradients, I get the following error when trying to visualize the results (after running cell 7):
```
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
in ()
1 # We can use `Visualizer` to draw the predictions on the image.
2 v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
----> 3 out = v.draw_instance_predictions(outputs[0]["instances"].to("cpu"))
4 cv2_imshow(out.get_image()[:, :, ::-1])
2 frames
/usr/local/lib/python3.7/dist-packages/detectron2/utils/visualizer.py in _convert_boxes(self, boxes)
1178 """
1179 if isinstance(boxes, Boxes) or isinstance(boxes, RotatedBoxes):
-> 1180 return boxes.tensor.numpy()
1181 else:
1182 return np.asarray(boxes)
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
```
## Workaround (run this before visualizing the results):
```
outputs[0]["instances"] = outputs[0]["instances"].to("cpu")
outputs[0]["instances"].pred_boxes.tensor = outputs[0]["instances"].pred_boxes.tensor.detach()
outputs[0]["instances"].scores = outputs[0]["instances"].scores.detach()
```
Contributor guide
Research direction
Reproduce the failure with the modified Colab cells and inspect detectron2/utils/visualizer.py at _convert_boxes, especially the boxes.tensor.numpy() call shown in the traceback. Verify visualization of gradient-attached inference outputs without the reported workaround, and confirm the existing workaround behavior remains valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100