Use TensorFlow Lite quantize RetinaFace get wrong result

1. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 18.04
• TensorFlow installation (pip package or built from source):docker pull
• TensorFlow library (version, if pip package or github SHA, if built from source):tensorflow:2.4.1-gpu

2. Code

this is my script which intend to convert float32 to int8 type:

import tensorflow as tf
import numpy as np
import cv2
import os

from nets.mobilenet025 import relu6
from nets.layers import UpsampleLike
from nets.retinaface_training import box_smooth_l1, conf_loss

save_folder = "./model_data/quantized"
model_path = "./model_data/RetinaFace.h5"

def representative_dataset_gen():
"""
# generate representative image dataset
:return:
"""
input_size = [640, 640]
dataset = "./imagelist.txt"
filename = open(dataset).read().split()
for i in range(len(filename)):
if os.path.exists(filename[i]):
print ("filename: ", i, filename[i])
orig_image = cv2.imread(filename[i])
rgb_image = cv2.cvtColor(orig_image, cv2.COLOR_BGR2RGB)
image_tensor = cv2.resize(rgb_image, dsize=tuple(input_size))
image_tensor = np.asarray(image_tensor / 255.0, dtype=np.float32)
image_tensor = image_tensor[np.newaxis, :]
yield [image_tensor]

if name == "main":
model = tf.keras.models.load_model(model_path, custom_objects={'relu6': relu6, 'UpsampleLike': UpsampleLike,
'_smooth_l1': box_smooth_l1, '_conf_loss': conf_loss})

export_dir = "./model_data/saved_model"
tf.saved_model.save(model, export_dir)
model = tf.saved_model.load(export_dir)
concrete_func = model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
concrete_func.inputs[0].set_shape([1, 640, 640, 3])
converter = tf.lite.TFLiteConverter.from_concrete_functions([concrete_func])

converter.experimental_new_converter = True
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset_gen
# Ensure that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.TFLITE_BUILTINS]
# Set the input and output tensors to int8 (APIs added in r2.3)
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_model_quant = converter.convert()

interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
input_type = interpreter.get_input_details()[0]['dtype']
print('input: ', input_type)
output_type = interpreter.get_output_details()[0]['dtype']
print('output: ', output_type)
# Save the quantized model:
tflite_model_quant_file = os.path.join(save_folder, "RetinaFace_quant.tflite")
open(tflite_model_quant_file, "wb").write(tflite_model_quant)

3. Failure after conversion

I followed the tflite spec write the above script, the conversion is successful, but the Model produces wrong results, the detected face have same conf score, the bbox location and landmask both is wrong..

Start with the conversion script, representative_dataset_gen, and the RetinaFace.h5 and imagelist.txt inputs; compare float and int8 outputs for the same images and inspect the interpreter's input and output details. Done means the cause of the incorrect confidence scores, bounding boxes, and landmarks is isolated and documented with a reproducible result.