tensorflow / tensorflow/models

confused with coco map meaning and can't get good result

Open
#10,337 0 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Oct 31, 2021.

models:research:odapi type:docs
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Prerequisites

Please answer the following question for yourself before submitting an issue.

  • I checked to make sure that this issue has not been filed already.

1. The entire URL of the documentation with the issue

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md#user-content-fn-1-389938ae008e4b49d16b1c9c0c403066

2. Describe the issue

I'm confused with the coco map metrics for "ssd_mobilenet_v1_coco" since coco has 12 evaluate metrics now, does it mean (AP) @IoU=0.50:0.95 or (AP) @IoU=0.50? Also, I have converted the "ssd_mobilenet_v1_coco" into tflite file without postprocess node, and then I decoded the two raw outputs by myself(all postprocess arguments were correctly taken from config file and the process was similar to "detection_postprocess.cc", but was written in python by myself). But the coco metrics I got seems not good. I'm not sure if it's my program bug, or using the default argument is not correct. How to reproduce the 21% result? Also, I have tried to generate a tflite file with the postprocess node, and I tried several inputs and found that the results of the 4 outputs are similar to what I have got by my python script.
This is my result below.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.184
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.307
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.192
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.017
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.157
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.372
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.180
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.242
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.242
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.026
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.206
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.491
In addition, I found that this issue is similar to https://github.com/tensorflow/models/issues/6636, and changing "nms_score_threshold" is not working(in fact, I have already noticed this while I'm reading sample config file and remarks of model zoo page, it helps increasing map actually, but 18.4% is the limitation, impossible to get 21% for me).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.