facebookresearch / facebookresearch/detectron2
confidence and iou threshold confusion
- Dominant language
- Python
- Stars
- 34.7k
- Forks
- 7.9k
- PR merge metrics
- No merged PRs in 30d
Description
Hello everyone,
This is kind of a long question with many parts, so bear with me.
Firstly, to clarify, is the following correct?
`SCORE_THRESH_TEST` = model confidence threshold (controls which predictions to keep based on model confidence)
`NMS_THRESH_TEST` = iou threshold (controls if predictions are true/false positives)
If so, I have a question regarding the values for each of these thresholds.
I was sifting through some research when I found that many papers used confidence thresholds of 0.001 and iou thresholds of 0.6 for evaluation purposes.
When I run the following code:
```
cfg.MODEL.WEIGHTS = f"{path_to_rcnn}/model_final.pth"
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.001 # confidence
cfg.MODEL.ROI_HEADS.NMS_THRESH_TEST = 0.6 # iou
predictor = DefaultPredictor(cfg)
evaluator = COCOEvaluator("test", output_dir="./output_eval")
val_loader = build_detection_test_loader(cfg, "test")
results = inference_on_dataset(predictor.model, val_loader, evaluator)
```
I get the following ap values: `AP: 59.00876493605871, AP50: 77.99360485252787` and the following images as predictions:


However, when I adjust the confidence and iou thresholds to something like
```
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.25 # confidence
cfg.MODEL.ROI_HEADS.NMS_THRESH_TEST = 0.45 # iou
```
I get the following ap values: `AP: 57.13751528250136, AP50: 75.08012744375405` and the following images as predictions:


What I am confused about is, wouldn't the first set of confidence and iou thresholds (0.001, 0.6) result in extremely low mAP due to a high number of false positives? Why does it actually have a higher mAP even though there are so many false positives?
Contributor guide
Research direction
Start by reading the definitions and evaluation flow for SCORE_THRESH_TEST and NMS_THRESH_TEST, then trace DefaultPredictor, COCOEvaluator, build_detection_test_loader, and inference_on_dataset. Compare how the two threshold sets affect predictions and reported AP, using the values and outputs in the issue. Done means the threshold semantics and the differing metrics are explained clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100