tensorflow / tensorflow/models

coco_detection_metrics and pascal_voc_detection_metrics mismatch

Open
#8,422 1 comment 0 reactions 1 assignee View on GitHub

@pkulzc is already working on this.

Since Apr 26, 2020.

models:research type:bug
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  • Mobile device (e.g., Pixel 4, Samsung Galaxy 10) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary):
  • TensorFlow version (use command below):
  • Python version:
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version:
  • GPU model and memory:

Please provide the entire URL of the model you are using?

https://github.com/tensorflow/models/tree/master/research/object_detection
Describe the current behavior
When I run eval with coco_detection_metrics I have AP (IOU=0.5)=0.41
When I run eval with pascal_voc_detection_metrics I have AP (IOU=0.5) = 0.86
Describe the expected behavior
This metrics must be equal
Code to reproduce the issue

config:
model {
faster_rcnn {
num_classes: 2
image_resizer {
keep_aspect_ratio_resizer {
min_dimension: 400
max_dimension: 400
}
}
feature_extractor {
type: 'faster_rcnn_resnet50'
first_stage_features_stride: 8
}
first_stage_anchor_generator {
grid_anchor_generator {
scales: [0.25, 0.5]
aspect_ratios: [0.5, 1.0]
height_stride: 8
width_stride: 8
}
}
first_stage_box_predictor_conv_hyperparams {
op: CONV
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
truncated_normal_initializer {
stddev: 0.01
}
}
}
first_stage_nms_score_threshold: 0.0
first_stage_nms_iou_threshold: 0.0
first_stage_localization_loss_weight: 2.0
first_stage_objectness_loss_weight: 1.0
initial_crop_size: 14
maxpool_kernel_size: 2
maxpool_stride: 2
second_stage_box_predictor {
mask_rcnn_box_predictor {
use_dropout: false
dropout_keep_probability: 1.0
fc_hyperparams {
op: FC
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
variance_scaling_initializer {
factor: 1.0
uniform: true
mode: FAN_AVG
}
}
}
}
}
second_stage_post_processing {
batch_non_max_suppression {
score_threshold: 0.0
iou_threshold: 0.0
}
score_converter: SOFTMAX
}
second_stage_localization_loss_weight: 2.0
second_stage_classification_loss_weight: 1.0
}
}

train_config: {
batch_size: 4
optimizer {
momentum_optimizer: {
learning_rate: {
manual_step_learning_rate {
initial_learning_rate: 0.0003
schedule {
step: 40000
learning_rate: .00003
}
schedule {
step: 510000
learning_rate: .000003
}
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}
gradient_clipping_by_norm: 10.0
fine_tune_checkpoint: ""
from_detection_checkpoint: true

num_steps: 2000
data_augmentation_options {
}

train_input_reader: {
tf_record_input_reader {
input_path: ""
}
label_map_path: ""

}

eval_config: {
num_examples: 30
max_evals: 1
num_visualizations: 30
eval_interval_secs: 900
use_moving_averages: false
metrics_set: "coco_detection_metrics"
}

eval_input_reader: {
tf_record_input_reader {
input_path: ""
}
label_map_path: ""
shuffle: false
num_readers: 1
}
Other info / logs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.