Couldn't convert ssd_efficientdet_d0_512x512_coco17_tpu-8 model to int8 tflite model

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [Yes] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [Yes] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [Yes] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/

2. Describe the bug

After training a model with config file "ssd_efficientdet_d0_512x512_coco17_tpu-8.config", executed the following steps:

1. Successfully export tensorflow model and tflite model with models/research/object_detection/exporter_main_v2.py
2. Successfully export detection SavedModel for tflite converstion with models/research/object_detection/export_tflite_graph_tf2.py.
3. Successfully converted the saved model from step 2 to tflite model with the following python code, and the generated model worked fine:

converter = tf.lite.TFLiteConverter.from_saved_model(args.model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.inference_input_type = tf.float32
converter.inference_output_type = tf.float32
tflite_model = converter.convert()

However, when I tried to generate an int8 tflite model with following code,
converter = tf.lite.TFLiteConverter.from_saved_model(args.model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.TFLITE_BUILTINS]
converter.inference_input_type = tf.float32
converter.inference_output_type = tf.float32
tflite_model = converter.convert()

it got a "core dumped" error message:
/home/xiangdong/experiment/8bit_tensorflow/facessd/export/ssd_efficientdet_d0_512x512_coco17_tpu-8_1229_lite/saved_model
2022-01-03 16:38:49.121464: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-01-03 16:38:54.890965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 22834 MB memory: -> device: 0, name: TITAN RTX, pci bus id: 0000:1a:00.0, compute capability: 7.5
2022-01-03 16:38:54.893600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 22834 MB memory: -> device: 1, name: TITAN RTX, pci bus id: 0000:1b:00.0, compute capability: 7.5
2022-01-03 16:38:54.895871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 22834 MB memory: -> device: 2, name: TITAN RTX, pci bus id: 0000:1d:00.0, compute capability: 7.5
2022-01-03 16:38:54.898263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 22834 MB memory: -> device: 3, name: TITAN RTX, pci bus id: 0000:1e:00.0, compute capability: 7.5
2022-01-03 16:38:54.900511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:4 with 22834 MB memory: -> device: 4, name: TITAN RTX, pci bus id: 0000:3d:00.0, compute capability: 7.5
2022-01-03 16:38:54.902890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:5 with 22834 MB memory: -> device: 5, name: TITAN RTX, pci bus id: 0000:3f:00.0, compute capability: 7.5
2022-01-03 16:38:54.905193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:6 with 22834 MB memory: -> device: 6, name: TITAN RTX, pci bus id: 0000:41:00.0, compute capability: 7.5
2022-01-03 16:38:54.907479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:7 with 22834 MB memory: -> device: 7, name: TITAN RTX, pci bus id: 0000:5e:00.0, compute capability: 7.5
2022-01-03 16:39:18.611278: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:363] Ignored output_format.
2022-01-03 16:39:18.611330: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:366] Ignored drop_control_dependency.
2022-01-03 16:39:18.611351: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:372] Ignored change_concat_input_ranges.
2022-01-03 16:39:18.612511: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /home/xiangdong/experiment/8bit_tensorflow/facessd/export/ssd_efficientdet_d0_512x512_coco17_tpu-8_1229_lite/saved_model
2022-01-03 16:39:18.849818: I tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-01-03 16:39:18.849888: I tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: /home/xiangdong/experiment/8bit_tensorflow/facessd/export/ssd_efficientdet_d0_512x512_coco17_tpu-8_1229_lite/saved_model
2022-01-03 16:39:19.723416: I tensorflow/cc/saved_model/loader.cc:210] Restoring SavedModel bundle.
2022-01-03 16:39:22.092031: I tensorflow/cc/saved_model/loader.cc:194] Running initialization op on SavedModel bundle at path: /home/xiangdong/experiment/8bit_tensorflow/facessd/export/ssd_efficientdet_d0_512x512_coco17_tpu-8_1229_lite/saved_model
2022-01-03 16:39:23.362740: I tensorflow/cc/saved_model/loader.cc:283] SavedModel load for tags { serve }; Status: success: OK. Took 4750232 microseconds.
2022-01-03 16:39:25.914092: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:237] disabling MLIR crash reproducer, set env var MLIR_CRASH_REPRODUCER_DIRECTORY to enable.
2022-01-03 16:39:40.347446: I tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1962] Estimated count of arithmetic ops: 6.784 G ops, equivalently 3.392 G MACs

Estimated count of arithmetic ops: 6.784 G ops, equivalently 3.392 G MACs
fully_quantize: 0, inference_type: 6, input_inference_type: 0, output_inference_type: 0
2022-01-03 16:42:13.951302: I tensorflow/compiler/mlir/lite/flatbuffer_export.cc:1962] Estimated count of arithmetic ops: 6.784 G ops, equivalently 3.392 G MACs

Estimated count of arithmetic ops: 6.784 G ops, equivalently 3.392 G MACs
./train.sh: line 82: 371560 Segmentation fault (core dumped) python3 convert_savedmodel_to_tflite.py --width=512 --height=512 --model_dir=${input_dir} --tflite_filename=${output_dir}/tf270_${model_name}_${today}.tflite

3. Steps to reproduce

A. The .sh that used for training and model conversion:
model_name=ssd_efficientdet_d0_512x512_coco17_tpu-8
today="1229"

rm -rf facessd/${model_name}${today}/model_dir
python3 models/research/object_detection/model_main_tf2.py
--num_workers=8
--checkpoint_every_n=5000
--pipeline_config_path=config/${model_name}.config
--model_dir=facessd/${model_name}${today}/model_dir
--alsologtostderr
--num_train_steps=300000
--sample_1_of_n_eval_examples=30

python3 models/research/object_detection/exporter_main_v2.py
--input_type image_tensor --pipeline_config_path config/${model_name}.config
--trained_checkpoint_dir facessd/${model_name}${today}/model_dir/
--output_directory facessd/export/${model_name}${today}/

python3 models/research/object_detection/export_tflite_graph_tf2.py
--pipeline_config_path /home/experiment/8bit_tensorflow/facessd/export/${model_name}${today}/pipeline.config
--trained_checkpoint_dir facessd/export/${model_name}${today}/checkpoint/
--output_directory facessd/export/${model_name}_${today}_lite/

output_dir=/home/experiment/8bit_tensorflow/facessd/export
input_dir=${output_dir}/${model_name}${today}lite/saved_model
python3 convert_savedmodel_to_tflite.py
--width=512
--height=512
--model_dir=${input_dir}
--tflite_filename=${output_dir}/tf270${model_name}${today}.tflite

B. The training config file:

model {
ssd {
inplace_batchnorm_update: true
freeze_batchnorm: false
num_classes: 1
add_background_class: false
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
use_matmul_gather: true
}
}
similarity_calculator {
iou_similarity {
}
}
encode_background_as_zeros: true
anchor_generator {
multiscale_anchor_generator {
min_level: 3
max_level: 7
anchor_scale: 4.0
aspect_ratios: [1.0, 2.0, 0.5]
scales_per_octave: 3
}
}
image_resizer {
fixed_shape_resizer {
height: 512
width: 512
}
}
box_predictor {
weight_shared_convolutional_box_predictor {
depth: 64
class_prediction_bias_init: -4.6
conv_hyperparams {
force_use_bias: true
activation: SWISH
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
random_normal_initializer {
stddev: 0.01
mean: 0.0
}
}
batch_norm {
scale: true
decay: 0.99
epsilon: 0.001
}
}
num_layers_before_predictor: 3
kernel_size: 3
use_depthwise: true
}
}
feature_extractor {
type: 'ssd_efficientnet-b0_bifpn_keras'
bifpn {
min_level: 3
max_level: 7
num_iterations: 3
num_filters: 64
}
conv_hyperparams {
force_use_bias: true
activation: SWISH
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
batch_norm {
scale: true,
decay: 0.99,
epsilon: 0.001,
}
}
}
loss {
classification_loss {
weighted_sigmoid_focal {
alpha: 0.25
gamma: 1.5
}
}
localization_loss {
weighted_smooth_l1 {
}
}
classification_weight: 1.0
localization_weight: 1.0
}
normalize_loss_by_num_matches: true
normalize_loc_loss_by_codesize: true
post_processing {
batch_non_max_suppression {
score_threshold: 1e-8
iou_threshold: 0.5
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SIGMOID
}
}
}

train_config: {
batch_size: 16
sync_replicas: true
startup_delay_steps: 0
replicas_to_aggregate: 8
use_bfloat16: true
num_steps: 300000
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
random_scale_crop_and_pad_to_square {
output_size: 512
scale_min: 0.1
scale_max: 2.0
}
}
optimizer {
momentum_optimizer: {
learning_rate: {
cosine_decay_learning_rate {
learning_rate_base: 8e-2
total_steps: 300000
warmup_learning_rate: .001
warmup_steps: 2500
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}
max_number_of_boxes: 100
unpad_groundtruth_tensors: false
}

train_input_reader: {
label_map_path: "tfrecord/head_label_map.pbtxt"
tf_record_input_reader {
input_path: "tfrecord/head_1024x512_train.tfrecord"
}
}

eval_config: {
metrics_set: "coco_detection_metrics"
use_moving_averages: false
batch_size: 1
}

eval_input_reader: {
label_map_path: "tfrecord/head_label_map.pbtxt"
shuffle: false
num_epochs: 1
tf_record_input_reader {
input_path: "tfrecord/head_1024x512_test.tfrecord"
}
}

graph_rewriter {
quantization {
delay: 2000
weight_bits: 8
activation_bits: 8
}
}

4. Expected behavior

That we can convert the trained model to a int8 tflite model.

5. Additional context

Tried using relu6 as activation instead of SWISH, result was the same.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• Mobile device name if the issue happens on a mobile device: PC
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): v2.7.0-rc1-69-gc256c071bb2 2.7.0
• Python version: 3.8.5
• Bazel version (if compiling from source): 3.4.1
• GCC/Compiler version (if compiling from source): Ubuntu 9.3.0-17ubuntu1~20.04
• CUDA/cuDNN version: Cuda 10.1, cudnn 8.1
• GPU model and memory: Titan RTX 24G