object detection: problem with class agnostic in SSD_mobilenet_v2 architecture

Hey,

I tried to use use_class_agnostic_nms: true in my pipeline.config. Has anyone used class agnostic in ssd architecture?

System information

• What is the top-level directory of the model you are using:
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): 1.14
• CUDA/cuDNN version: 10.2
• GPU model and memory: GTX 1080Ti

Describe the problem

When I set use_class_agnostic_nms: true in my pipeline config, there is a problem with evaluation process of the model, the exact problem is visible in the logs below. Any ideas how to solve the problem?

Source code / logs

Error log:

[2020-02-14 09:48:59,116] {{taskinstance.py:1058}} ERROR - TensorArray dtype is float but Op is trying to write dtype int64.
	 [[node Postprocessor/BatchMultiClassNonMaxSuppression/map/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py:51) ]]

Original stack trace for 'Postprocessor/BatchMultiClassNonMaxSuppression/map/while/TensorArrayWrite_2/TensorArrayWriteV3':
  File "/bin/airflow", line 37, in <module>
    args.func(args)
  File "/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "/lib/python3.7/site-packages/airflow/bin/cli.py", line 551, in run
    _run(args, dag, ti)
  File "/lib/python3.7/site-packages/airflow/bin/cli.py", line 469, in _run
    pool=args.pool,
  File "/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/airflow/dags/airflow_training_dag/src/task_operator_utils.py", line 398, in evaluate_last_experiment
    local_paths["metrics"], min_confidence=config_dict["min_confidence"]
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/model_training/experiment_evaluator.py", line 43, in calculate_metrics
    self.model_path, self.classes_file_name, min_confidence=min_confidence
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/metrics_calculator.py", line 247, in get_preds_from_model
    ps = TFModelPredictionService(model_path, classes_file, min_confidence)
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/services/tf_model_pred_service.py", line 19, in __init__
    self.model = self.load_model(model_file, classes_file, min_confidence)
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/services/tf_model_pred_service.py", line 24, in load_model
    model = TensorFlowModel(model_base_path, graph_name, classess_file, min_confidence)
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py", line 23, in __init__
    self.model = self.load_model()
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py", line 51, in load_model
    tf.import_graph_def(graphDef, name="")
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def
    _ProcessNewOps(graph)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3751, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()
Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray dtype is float but Op is trying to write dtype int64.
	 [[{{node Postprocessor/BatchMultiClassNonMaxSuppression/map/while/TensorArrayWrite_2/TensorArrayWriteV3}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/airflow_training_dag/src/task_operator_utils.py", line 398, in evaluate_last_experiment
    local_paths["metrics"], min_confidence=config_dict["min_confidence"]
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/assecobs/mate/model_training/experiment_evaluator.py", line 43, in calculate_metrics
    self.model_path, self.classes_file_name, min_confidence=min_confidence
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/metrics_calculator.py", line 251, in get_preds_from_model
    **query_params
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/services/prediction_service.py", line 28, in get_preds_for_batch
    **query_params)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/services/tf_model_pred_service.py", line 46, in get_preds
    preds = self.model.predict(img_object.img)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py", line 104, in predict
    (boxes, scores, labels) = self.predict_from_image(image)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py", line 92, in predict_from_image
    feed_dict={imageTensor: image},
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray dtype is float but Op is trying to write dtype int64.
	 [[node Postprocessor/BatchMultiClassNonMaxSuppression/map/while/TensorArrayWrite_2/TensorArrayWriteV3 (defined at /airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py:51) ]]

Original stack trace for 'Postprocessor/BatchMultiClassNonMaxSuppression/map/while/TensorArrayWrite_2/TensorArrayWriteV3':
  File "/bin/airflow", line 37, in <module>
    args.func(args)
  File "/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "/lib/python3.7/site-packages/airflow/bin/cli.py", line 551, in run
    _run(args, dag, ti)
  File "/lib/python3.7/site-packages/airflow/bin/cli.py", line 469, in _run
    pool=args.pool,
  File "/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
  File "/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/airflow/dags/airflow_training_dag/src/task_operator_utils.py", line 398, in evaluate_last_experiment
    local_paths["metrics"], min_confidence=config_dict["min_confidence"]
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/model_training/experiment_evaluator.py", line 43, in calculate_metrics
    self.model_path, self.classes_file_name, min_confidence=min_confidence
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/metrics_calculator.py", line 247, in get_preds_from_model
    ps = TFModelPredictionService(model_path, classes_file, min_confidence)
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/services/tf_model_pred_service.py", line 19, in __init__
    self.model = self.load_model(model_file, classes_file, min_confidence)
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/services/tf_model_pred_service.py", line 24, in load_model
    model = TensorFlowModel(model_base_path, graph_name, classess_file, min_confidence)
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py", line 23, in __init__
    self.model = self.load_model()
  File "/airflow/.local/lib/python3.7/site-packages/assecobs/mate/metrics/models/tensorflow_model.py", line 51, in load_model
    tf.import_graph_def(graphDef, name="")
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 443, in import_graph_def
    _ProcessNewOps(graph)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/importer.py", line 236, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3751, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3751, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3641, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/airflow/.local/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

My piepline.config:

model {
  ssd {
    num_classes: 61
    image_resizer {
      fixed_shape_resizer {
        height: 1024
        width: 1024
      }
    }
    feature_extractor {
      type: "ssd_mobilenet_v2"
      depth_multiplier: 1.0
      min_depth: 16
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 3.9999998989515007e-05
          }
        }
        initializer {
          truncated_normal_initializer {
            mean: 0.0
            stddev: 0.029999999329447746
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.9997000098228455
          center: true
          scale: true
          epsilon: 0.0010000000474974513
          train: true
        }
      }
    }
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    box_predictor {
      convolutional_box_predictor {
        conv_hyperparams {
          regularizer {
            l2_regularizer {
              weight: 3.9999998989515007e-05
            }
          }
          initializer {
            truncated_normal_initializer {
              mean: 0.0
              stddev: 0.029999999329447746
            }
          }
          activation: RELU_6
          batch_norm {
            decay: 0.9997000098228455
            center: true
            scale: true
            epsilon: 0.0010000000474974513
            train: true
          }
        }
        min_depth: 0
        max_depth: 0
        num_layers_before_predictor: 0
        use_dropout: false
        dropout_keep_probability: 0.800000011920929
        kernel_size: 1
        box_code_size: 4
        apply_sigmoid_to_scores: false
      }
    }
    anchor_generator {
      ssd_anchor_generator {
        num_layers: 6
        min_scale: 0.20000000298023224
        max_scale: 0.949999988079071
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 3.0
        aspect_ratios: 0.33329999446868896
      }
    }
    post_processing {
      batch_non_max_suppression {
        score_threshold: 9.99999993922529e-09
        iou_threshold: 0.6000000238418579
        max_detections_per_class: 100
        max_total_detections: 100
	use_class_agnostic_nms: true
      }
      score_converter: SIGMOID
    }
    normalize_loss_by_num_matches: true
    loss {
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      classification_loss {
        weighted_sigmoid {
        }
      }
      hard_example_miner {
        num_hard_examples: 3000
        iou_threshold: 0.9900000095367432
        loss_type: CLASSIFICATION
        max_negatives_per_positive: 3
        min_negatives_per_image: 3
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
  }
}
train_config {
  batch_size: 2
  data_augmentation_options {
    ssd_random_crop {
    }
  }
  data_augmentation_options {
    autoaugment_image {
    }
  }
  optimizer {
    rms_prop_optimizer {
      learning_rate {
        exponential_decay_learning_rate {
          initial_learning_rate: 2.0999999046325684
          decay_steps: 2500
          decay_factor: 0.800000011920929
        }
      }
      momentum_optimizer_value: 0.8999999761581421
      decay: 0.8999999761581421
      epsilon: 1.0
    }
  }
  fine_tune_checkpoint: "/experiment/model.ckpt-162590"
  num_steps: 201180
  fine_tune_checkpoint_type: "detection"
}