facebookresearch / facebookresearch/detectron2

Visualizer: cv2.findContours() produces overlapping prediction masks

Open
#3,515 4 comments 0 reactions 0 assignees View on GitHub
contributions welcome enhancement
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

## Instructions To Reproduce the Issue:
Train any model on a very small number of images before testing. When using Visualizer from detectron2.utils.visualizer, when visualising test images you will see (on occasion) overlapping segmentation masks. Such as:

![image](https://user-images.githubusercontent.com/87654835/134494069-7d76347d-a0cc-4335-87ed-6e74754161dc.png)

Note that the small bits are not holes, they are masks overlapping themselves.

## Expected behavior:
All images visualised to not have any overlaps. Like so:
![image](https://user-images.githubusercontent.com/87654835/134495043-9930d1e6-91a3-4260-9c26-4f54c46ed87e.png)

## Cause and possible solution
The issue originates from the mask_to_polygon function [here](https://github.com/facebookresearch/detectron2/blob/main/detectron2/utils/visualizer.py#L116) function. Specifically, the cv2.findContours() can erroneously output contours that are subsets of one another. In order to achieve the expected behaviour above, I overwrote the mask_to_polygon function to remove any contours that are subsets of others. The code is below. It is fast (doesn't add much time to the inference at all <0.1seconds per image), but clearly it is not very efficient currently and could be improved.

```
def mask_to_polygons(self, mask):
# cv2.RETR_CCOMP flag retrieves all the contours and arranges them to a 2-level
# hierarchy. External contours (boundary) of the object are placed in hierarchy-1.
# Internal contours (holes) are placed in hierarchy-2.
# cv2.CHAIN_APPROX_NONE flag gets vertices of polygons from contours.
mask = np.ascontiguousarray(mask) # some versions of cv2 does not support incontiguous arr
res, hierarchy = cv2.findContours(mask.astype("uint8"), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_NONE)

if hierarchy is None: # empty mask
return [], False

#TODO speed this search up
new_polys = res.copy()
indices_to_keep = list(range(len(res)))
for i,r1 in enumerate(res):
for j,r2 in enumerate(res):
if i == j:
continue
if np.in1d(r2.ravel(), r1.ravel()).all():
if len(r2) > len(r1):
if i in indices_to_keep:
indices_to_keep.remove(i)
elif len(r1) > len(r2):
if j in indices_to_keep:
indices_to_keep.remove(j)

res = [val for i,val in enumerate(new_polys) if i in indices_to_keep]

has_holes = (hierarchy.reshape(-1, 4)[:, 3] >= 0).sum() > 0
res = [x.flatten() for x in res]
# These coordinates from OpenCV are integers in range [0, W-1 or H-1].
# We add 0.5 to turn them into real-value coordinate space. A better solution
# would be to first +0.5 and then dilate the returned polygon by 0.5.
res = [x + 0.5 for x in res if len(x) >= 6]
return res, has_holes
```

## Environment:
Detectron2 basic configuration

Contributor guide

Open the contributing guide

Research direction

Start in detectron2/utils/visualizer.py at the mask_to_polygon function around line 116, then reproduce the issue by training a model on very few images and visualizing test images. Inspect how cv2.findContours returns overlapping or nested contours and compare the output with the expected non-overlapping masks shown in the issue. Done means the visualizer no longer produces self-overlapping prediction masks while preserving valid holes.

Written by the indexing model from the issue text.

Assessment

Tech stack
opencv, python
Domain
computer-vision
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.