tensorflow / tensorflow/models
Mismatched mAP of TF2 object detection modelzoo
@pkulzc is already working on this.
Since Aug 19, 2020.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
Prerequisites
Please answer the following question for yourself before submitting an issue.
- I checked to make sure that this issue has not been filed already.
1. The entire URL of the documentation with the issue
2. Describe the issue
I use TF-2.3.0 and follow your instruction to set up the environment for OD API. I try to test your models of TF2.X from the detection modelzoo, only to find my mAPs seem higher than the reported on the table. I use the 8059 images from your given list to create tfrecord for evaluation.
Could you check if the mAPs of the TF2-OD-Modelzoo are given with your default settings (score_thresh, nms parameters, etc.) in the pipeline.config ? My results are listed in the following table for comparison.
(Another problem: the input_size in your SSD-MobileNet-v2-320x320 pipeline.config is 300x300 instead of 320x320)
I'll appreciate it if you could help to check the issue.
| model | Claimed mAP | my results |
|---|---|---|
| SSD MobileNet v2 320x320 | 20.2 | Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.244 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.413 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.251 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.014 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.113 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.463 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.242 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.363 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.382 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.278 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.651 |
| SSD MobileNet V1 FPN 640x640 | 29.1 | Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.335 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.521 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.366 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.094 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.308 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.470 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.297 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.477 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.515 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.222 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.519 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.668 |
| SSD ResNet50 V1 FPN 640x640 (RetinaNet50) | 34.3 | Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.402 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.592 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.446 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.126 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.384 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.556 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.334 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.530 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.568 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.264 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.582 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.726 |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.