Issues while loading custom trained SSD MobileNet V2 in tflite format (using tflite module supported by Flutter)

1. The entire URL of the file you are using

http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz

2. Describe the bug

Custom trained SSD MoblieNet V2 raises exception on all platforms (iOS and Android). Mobile app has been developed using Flutter with tflite 1.1.0 module. Neural network works fine (detects objects) when it is imported as .pb file (tested using models/research/object_detection/object_detection_tutorial.ipynb).

3. Steps to reproduce

I have followed

python model_main.py --alsologtostderr --model_dir=training/ --pipeline_config_path=training1/pipeline.config

python export_tflite_ssd_graph.py --pipeline_config_path training/pipeline.config --trained_checkpoint_prefix training/model.ckpt-2000 --output_directory export

tflite_convert --graph_def_file=export/tflite_graph.pb --output_file=export/model.tflite --output_format=TFLITE --input_arrays=image_tensor --input_shapes=1,300,300,3 --inference_type=FLOAT --output_arrays="TFLite_Detection_PostProcess,TFLite_Detection_PostProcess:1,TFLite_Detection_PostProcess:2,TFLite_Detection_PostProcess:3" --allow_custom_ops

pipeline.config file

# SSD with Mobilenet v2 configuration for MSCOCO Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.

model {
  ssd {
    num_classes: 1
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.5
        unmatched_threshold: 0.5
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    anchor_generator {
      ssd_anchor_generator {
        num_layers: 6
        min_scale: 0.2
        max_scale: 0.95
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 3.0
        aspect_ratios: 0.3333
      }
    }
    image_resizer {
      fixed_shape_resizer {
        height: 300
        width: 300
      }
    }
    box_predictor {
      convolutional_box_predictor {
        min_depth: 0
        max_depth: 0
        num_layers_before_predictor: 0
        use_dropout: false
        dropout_keep_probability: 0.8
        kernel_size: 1
        box_code_size: 4
        apply_sigmoid_to_scores: false
        conv_hyperparams {
          activation: RELU_6,
          regularizer {
            l2_regularizer {
              weight: 0.00004
            }
          }
          initializer {
            truncated_normal_initializer {
              stddev: 0.03
              mean: 0.0
            }
          }
          batch_norm {
            train: true,
            scale: true,
            center: true,
            decay: 0.9997,
            epsilon: 0.001,
          }
        }
      }
    }
    feature_extractor {
      type: 'ssd_mobilenet_v2'
      min_depth: 16
      depth_multiplier: 1.0
      conv_hyperparams {
        activation: RELU_6,
        regularizer {
          l2_regularizer {
            weight: 0.00004
          }
        }
        initializer {
          truncated_normal_initializer {
            stddev: 0.03
            mean: 0.0
          }
        }
        batch_norm {
          train: true,
          scale: true,
          center: true,
          decay: 0.9997,
          epsilon: 0.001,
        }
      }
    }
    loss {
      classification_loss {
        weighted_sigmoid {
        }
      }
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      hard_example_miner {
        num_hard_examples: 3000
        iou_threshold: 0.99
        loss_type: CLASSIFICATION
        max_negatives_per_positive: 3
        min_negatives_per_image: 3
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    normalize_loss_by_num_matches: true
    post_processing {
      batch_non_max_suppression {
        score_threshold: 1e-8
        iou_threshold: 0.6
        max_detections_per_class: 100
        max_total_detections: 100
      }
      score_converter: SIGMOID
    }
  }
}

train_config: {
  batch_size: 24
  optimizer {
    rms_prop_optimizer: {
      learning_rate: {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.004
          decay_steps: 800720
          decay_factor: 0.95
        }
      }
      momentum_optimizer_value: 0.9
      decay: 0.9
      epsilon: 1.0
    }
  }
  fine_tune_checkpoint: "pre-trained-model/model.ckpt"
  fine_tune_checkpoint_type:  "detection"
  # Note: The below line limits the training process to 200K steps, which we
  # empirically found to be sufficient enough to train the pets dataset. This
  # effectively bypasses the learning rate schedule (the learning rate will
  # never decay). Remove the below line to train indefinitely.
  num_steps: 2000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    ssd_random_crop {
    }
  }
}

train_input_reader: {
  tf_record_input_reader {
    input_path: "annotations/test.record"
  }
  label_map_path: "annotations/label_map.pbtxt"
}

eval_config: {
  num_examples: 200
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 10
}

eval_input_reader: {
  tf_record_input_reader {
    input_path: "annotations/train.record"
  }
  label_map_path: "annotations/label_map.pbtxt"
  shuffle: false
  num_readers: 1
}

then getting this error while loading tflite model in this section:

import 'package:flutter/material.dart';
import 'package:shopping_assistant/ui/app.dart';
import 'package:tflite/tflite.dart';

void main() async {
  await initializeModel();
  runApp(ShoppingAssistantApp());
}

Future<String> initializeModel() async {
  WidgetsFlutterBinding.ensureInitialized();

  return await Tflite.loadModel(
    model: 'assets/model/model.tflite',
    labels: 'assets/model/labels.txt',
    numThreads: 2
  );
}

Error message:

2020-07-01 22:00:08.321686+0200 Runner[567:67963] Metal API Validation Enabled
2020-07-01 22:00:09.151457+0200 Runner[567:68177] flutter: Observatory listening on http://127.0.0.1:50364/OB3EB-AMXXw=/
Loaded model 1resolved reporter2020-07-01 22:00:11.353310+0200 Runner[567:67963] * Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '* -[__NSDictionaryM setObject:forKey:]: key cannot be nil'
* First throw call stack:
(0x1870b4794 0x186dd6bcc 0x18710a2dc 0x1871138f0 0x186f90ff0 0x100af0fc4 0x100af2be4 0x100aee898 0x100e0a338 0x100e0b730 0x100e19710 0x1870327fc 0x18702d6d0 0x18702cce8 0x19117738c 0x18b15b444 0x100ae5ce0 0x186eb48f0)
libc++abi.dylib: terminating with uncaught exception of type NSException
(Recorded stack frame)

on Android platform:

4. Expected behavior

I expect that MobileNet should be loaded errorless and detect selected objects.

5. Additional context

I have found that this issue also occurs while loading other NN that are not quantized. However model_main.py goes OOM when training quantized SSD MobileNet V2 - with normal version works fine.

ader.cc:42] Successfully opened dynamic library libcudnn.so.7
INFO:tensorflow:loss = 13.815958, step = 0
I0706 00:42:46.171965 140055071561536 basic_session_run_hooks.py:262] loss = 13.815958, step = 0
2020-07-06 00:43:11.607559: E tensorflow/stream_executor/cuda/cuda_driver.cc:890] failed to alloc 4294967296 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-07-06 00:43:11.609183: W ./tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 4294967296
2020-07-06 00:43:11.610017: E tensorflow/stream_executor/cuda/cuda_driver.cc:890] failed to alloc 3865470464 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-07-06 00:43:11.610033: W ./tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 3865470464
2020-07-06 00:43:11.610063: E tensorflow/stream_executor/cuda/cuda_driver.cc:890] failed to alloc 3478923264 bytes on host: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-07-06 00:43:11.610073: W ./tensorflow/core/common_runtime/gpu/gpu_host_allocator.h:44] could not allocate pinned host memory of size: 3478923264
Killed

Also approach to convert (using tflite_convert) not trained quantized SSD MobileNet fails - returns the same exception like normal NN.

Only quantized SSD MobileNet downloaded from: https://zenodo.org/record/3361502#.XwJYdpaxU5k works as expected.

It is worth mention, that I have inspected those NN using Netron and I could not have found difference in their structures (tflite files).

6. System information

• OS Platform and Distribution: Linux 18.04
• Mobile device name if the issue happens on a mobile device: iPhone 6s
• TensorFlow version (use command below): 1.14
• Python version: 3.6.9
• CUDA/cuDNN version: 9.2
• GPU model and memory: NVIDIA GeForce GTX 1050 Ti Mobile 4096 MB GDDR5
• Flutter: 1.20.0-3.0.pre.124
• Dart: 2.9.0 (build 2.9.0-20.0.dev f8ff12008e)