facebookresearch / facebookresearch/detectron2

confidence and iou threshold confusion

Open
#4,990 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

Hello everyone,

This is kind of a long question with many parts, so bear with me.

Firstly, to clarify, is the following correct?

`SCORE_THRESH_TEST` = model confidence threshold (controls which predictions to keep based on model confidence)
`NMS_THRESH_TEST` = iou threshold (controls if predictions are true/false positives)

If so, I have a question regarding the values for each of these thresholds.

I was sifting through some research when I found that many papers used confidence thresholds of 0.001 and iou thresholds of 0.6 for evaluation purposes.

When I run the following code:
```
cfg.MODEL.WEIGHTS = f"{path_to_rcnn}/model_final.pth"

cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.001 # confidence
cfg.MODEL.ROI_HEADS.NMS_THRESH_TEST = 0.6 # iou

predictor = DefaultPredictor(cfg)

evaluator = COCOEvaluator("test", output_dir="./output_eval")
val_loader = build_detection_test_loader(cfg, "test")
results = inference_on_dataset(predictor.model, val_loader, evaluator)
```

I get the following ap values: `AP: 59.00876493605871, AP50: 77.99360485252787` and the following images as predictions:

![Screenshot 2023-06-04 at 10 17 22 AM](https://github.com/facebookresearch/detectron2/assets/76795078/48bb26f2-6aa6-4ea8-b962-e21a60675c0d)

![Screenshot 2023-06-04 at 10 17 37 AM](https://github.com/facebookresearch/detectron2/assets/76795078/ab6f60c4-24bc-4a06-b043-494e7fcb1ee6)

However, when I adjust the confidence and iou thresholds to something like

```
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.25 # confidence
cfg.MODEL.ROI_HEADS.NMS_THRESH_TEST = 0.45 # iou
```

I get the following ap values: `AP: 57.13751528250136, AP50: 75.08012744375405` and the following images as predictions:

![Screenshot 2023-06-04 at 10 20 44 AM](https://github.com/facebookresearch/detectron2/assets/76795078/487807fe-a153-4c5e-8793-62136e9067b7)

![Screenshot 2023-06-04 at 10 20 12 AM](https://github.com/facebookresearch/detectron2/assets/76795078/0665c5df-da58-494a-a893-e1c954f11ae1)

What I am confused about is, wouldn't the first set of confidence and iou thresholds (0.001, 0.6) result in extremely low mAP due to a high number of false positives? Why does it actually have a higher mAP even though there are so many false positives?

Contributor guide

Open the contributing guide

Research direction

Start by reading the definitions and evaluation flow for SCORE_THRESH_TEST and NMS_THRESH_TEST, then trace DefaultPredictor, COCOEvaluator, build_detection_test_loader, and inference_on_dataset. Compare how the two threshold sets affect predictions and reported AP, using the values and outputs in the issue. Done means the threshold semantics and the differing metrics are explained clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.