Integer tflite model slow than float32 model

1. System information

• quantized integer tflite model in Window10(without Nvidia GPU)
• TensorFlow installation method pip installed tensorflow-gpu-2.2.2 and tensorflow-2.2.2.
• try to run quantized model in mobile phone (Mi 9 with built-in GPU and DSP).
• MI 9: android 9
• nnapi-delegate tool version: 2.5.0
• gpu-delegate tool version: 2.5.0
• android.tools.build:gradle: 4.1.3
• compile Sdk version : 29 android 10
• build tools version: 29.0.2
• NDK version: 21.1.6352462

2. Code

    # part of my script
    try:
        keras_model = tf.keras.models.load_model(save_model_dir)
        # Normalize the input image so that each pixel value is between 0 to 1.
        converter = tf.lite.TFLiteConverter.from_keras_model(keras_model)
        converter.optimizations = [tf.lite.Optimize.DEFAULT]
        if INTEGER_QUANTIZATON:
            converter.representative_dataset = representative_data_gen1
            # Ensure that if any ops can't be quantized, the converter throws an error
            converter.representative_dataset = tf.lite.RepresentativeDataset(representative_data_gen1)
            converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
            # Set the input and output tensors to uint8 (APIs added in r2.3)
            converter.inference_input_type = tf.int8
            converter.inference_output_type = tf.int8
        tflite_model_quant = converter.convert()
        open(quantized_save_path, "wb").write(tflite_model_quant)

Option A: Reference colab notebooks

1. Reference url:
   https://www.tensorflow.org/lite/performance/post_training_integer_quant

3. Failure after conversion

I try to quantify the model to increase the speed of inference. The float32 tflite model work well on mobile gpu(mi 9) and DSP(mi 9), BUT afer integer quantized to int8 model, the int8 tflite inference slow down on gpu and cannot work on DSP.
floate32 tflite model
size: 86 KB
after quantized to integer model :
size: 38.3KB

5. Android studio logs

2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x7c29573cd0
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x0 0000007ff5514480 x1 0000007b8f610804 x2 0000000000000008 x3 0000007ff5514488
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x4 0000007b8f61080c x5 0000000000000000 x6 0000000200000004 x7 0000000200000004
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x8 0000007c29573cd0 x9 0000000000000000 x10 0000007ff5514488 x11 0000000000000002
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x12 0000000000038400 x13 00000000000383f8 x14 0000007b898e1010 x15 0000007b898e0f70
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x16 0000007b8fc171e8 x17 0000007c2afbb2c0 x18 0000000000000008 x19 0000007ff5514500
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x20 0000007ff5514480 x21 0000000000000008 x22 0000007c2edb95e0 x23 0000007c2edb95e0
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x24 0000007b8f6580e0 x25 0000000000000002 x26 0000007b8f665238 x27 0000007b8f6651c8
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: x28 000000000004b000 x29 0000007ff55144e0
2021-08-19 12:53:22.484 7948-7948/? A/DEBUG: sp 0000007ff5514470 lr 0000007b8facafe0 pc 0000007b8facb088
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: backtrace:
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #00 pc 00000000000f9088 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/base.apk (offset 0x25a000)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #01 pc 00000000000f9134 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/base.apk (offset 0x25a000)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #02 pc 00000000001c2be8 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/base.apk (offset 0x25a000)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #03 pc 00000000001c6404 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/base.apk (offset 0x25a000)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #04 pc 0000000000002f10 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/base.apk (offset 0x254000) (srdemo::SuperResolution::DoSuperResolution(signed char*)+292)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #05 pc 00000000000021a8 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/base.apk (offset 0x254000) (Java_com_example_srdemo_render_JniLib_SuperResolutionFromJNI+136)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #06 pc 0000000000046a50 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/oat/arm64/base.odex (offset 0x45000) (com.example.srdemo.render.JniLib.SuperResolutionFromJNI+160)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #07 pc 00000000000c17b4 /data/app/com.example.srdemo-iDI0OLee27Jcr3JbKsYvqg==/oat/arm64/base.odex (offset 0x45000) (com.example.srdemo.render.JniLib.GetYResult+116)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #08 pc 000000000000b584 /dev/ashmem/dalvik-jit-code-cache (deleted) (com.example.srdemo.render.SRGLSurfaceView.onPreviewFrame+404)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #09 pc 00000000000110a0 /dev/ashmem/dalvik-jit-code-cache (deleted) (com.example.srdemo.camera.CameraProxy.onPreviewFrame+160)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #10 pc 0000000000557388 /system/lib64/libart.so (art_quick_invoke_stub+584)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #11 pc 00000000000cfcc8 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #12 pc 0000000000280338 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #13 pc 000000000027a34c /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+968)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #14 pc 0000000000527444 /system/lib64/libart.so (MterpInvokeInterface+1392)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #15 pc 0000000000549b94 /system/lib64/libart.so (ExecuteMterpImpl+14740)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #16 pc 00000000005a6a3a /system/framework/boot-framework.vdex (android.hardware.Camera$EventHandler.handleMessage+542)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #17 pc 0000000000254050 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.3375396565+488)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #18 pc 0000000000259b44 /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #19 pc 000000000027a330 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+940)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #20 pc 00000000005264c8 /system/lib64/libart.so (MterpInvokeVirtual+588)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #21 pc 0000000000549994 /system/lib64/libart.so (ExecuteMterpImpl+14228)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #22 pc 0000000000bb4c42 /system/framework/boot-framework.vdex (android.os.Handler.dispatchMessage+42)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #23 pc 0000000000254050 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.3375396565+488)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #24 pc 0000000000259b44 /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #25 pc 000000000027a330 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+940)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #26 pc 00000000005264c8 /system/lib64/libart.so (MterpInvokeVirtual+588)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #27 pc 0000000000549994 /system/lib64/libart.so (ExecuteMterpImpl+14228)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #28 pc 0000000000bc781a /system/framework/boot-framework.vdex (android.os.Looper.loop+416)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #29 pc 0000000000254050 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.3375396565+488)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #30 pc 0000000000259b44 /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #31 pc 000000000027a330 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+940)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #32 pc 00000000005279cc /system/lib64/libart.so (MterpInvokeStatic+204)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #33 pc 0000000000549b14 /system/lib64/libart.so (ExecuteMterpImpl+14612)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #34 pc 0000000000421b46 /system/framework/boot-framework.vdex (android.app.ActivityThread.main+214)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #35 pc 0000000000254050 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.3375396565+488)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #36 pc 0000000000516d7c /system/lib64/libart.so (artQuickToInterpreterBridge+1020)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #37 pc 00000000005604fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #38 pc 000000000055764c /system/lib64/libart.so (art_quick_invoke_static_stub+604)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #39 pc 00000000000cfce8 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+232)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #40 pc 000000000045dd7c /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #41 pc 000000000045f7d0 /system/lib64/libart.so (art::InvokeMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jobject*, _jobject*, unsigned long)+1440)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #42 pc 00000000003ef398 /system/lib64/libart.so (art::Method_invoke(_JNIEnv*, _jobject*, _jobject*, _jobjectArray*)+52)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #43 pc 000000000078eed4 /system/framework/arm64/boot-core-oj.oat (offset 0x2dc000) (java.lang.Class.getDeclaredMethodInternal [DEDUPED]+180)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #44 pc 0000000000557388 /system/lib64/libart.so (art_quick_invoke_stub+584)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #45 pc 00000000000cfcc8 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #46 pc 0000000000280338 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #47 pc 000000000027a34c /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+968)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #48 pc 00000000005264c8 /system/lib64/libart.so (MterpInvokeVirtual+588)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #49 pc 0000000000549994 /system/lib64/libart.so (ExecuteMterpImpl+14228)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #50 pc 0000000001284a64 /system/framework/boot-framework.vdex (com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run+22)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #51 pc 0000000000254050 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.3375396565+488)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #52 pc 0000000000516d7c /system/lib64/libart.so (artQuickToInterpreterBridge+1020)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #53 pc 00000000005604fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #54 pc 000000000245d5dc /system/framework/arm64/boot-framework.oat (offset 0xa28000) (com.android.internal.os.ZygoteInit.main+2028)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #55 pc 000000000055764c /system/lib64/libart.so (art_quick_invoke_static_stub+604)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #56 pc 00000000000cfce8 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+232)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #57 pc 000000000045dd7c /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #58 pc 000000000045d9dc /system/lib64/libart.so (art::InvokeWithVarArgs(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, std::__va_list)+424)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #59 pc 0000000000362cd8 /system/lib64/libart.so (art::JNI::CallStaticVoidMethodV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+652)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #60 pc 00000000000b2884 /system/lib64/libandroid_runtime.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+116)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #61 pc 00000000000b53f4 /system/lib64/libandroid_runtime.so (android::AndroidRuntime::start(char const*, android::Vectorandroid::String8 const&, bool)+924)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #62 pc 0000000000002528 /system/bin/app_process64 (main+2012)
2021-08-19 12:53:22.667 7948-7948/? A/DEBUG: #63 pc 00000000000c888c /system/lib64/libc.so (__libc_init+88)
2021-08-19 12:53:22.756 7948-7948/? E/crash_dump64: cannot open libmiuindbg.so: No such file or directory
2021-08-19 12:53:22.760 985-985/? E//system/bin/tombstoned: Tombstone written to: /data/tombstones/tombstone_07
2021-08-19 12:53:22.828 1410-1941/? E/InputDispatcher: channel 'ad25ed com.example.srdemo/com.example.srdemo.MainActivity (server)' ~ Channel is unrecoverably broken and will be disposed!
2021-08-19 12:53:22.845 930-25654/? E/Camera2Client: notifyError: Error condition 0 reported by HAL, requestId -1
2021-08-19 12:53:22.869 783-870/? E/BufferQueueProducer: [SurfaceView - com.example.srdemo/com.example.srdemo.MainActivity#1] queueBuffer: BufferQueue has been abandoned
2021-08-19 12:53:22.870 930-1229/? E/Surface: queueBuffer: error queuing buffer to SurfaceTexture, -19
2021-08-19 12:53:22.879 783-870/? E/BufferQueueProducer: [SurfaceView - com.example.srdemo/com.example.srdemo.MainActivity#1] cancelBuffer: BufferQueue has been abandoned
2021-08-19 12:53:22.911 783-1273/? E/BufferQueueProducer: [SurfaceView - com.example.srdemo/com.example.srdemo.MainActivity#1] cancelBuffer: BufferQueue has been abandoned
2021-08-19 12:53:22.912 783-1273/? E/BufferQueueProducer: [SurfaceView - com.example.srdemo/com.example.srdemo.MainActivity#1] cancelBuffer: BufferQueue has been abandoned
2021-08-19 12:53:22.912 783-1273/? E/BufferQueueProducer: [SurfaceView - com.example.srdemo/com.example.srdemo.MainActivity#1] cancelBuffer: BufferQueue has been abandoned
2021-08-19 12:53:22.912 783-1273/? E/BufferQueueProducer: [SurfaceView - com.example.srdemo/com.example.srdemo.MainActivity#1] cancelBuffer: BufferQueue has been abandoned
2021-08-19 12:53:23.189 6722-6722/? E/Launcher: changeViewByFsGestureState, view=FitSystemWindowView, alpha=1.0, scale=1.0
2021-08-19 12:53:23.189 6722-6722/? E/Launcher: changeViewByFsGestureState, view=ShortcutMenuLayer, alpha=1.0, scale=1.0
2021-08-19 12:53:32.851 722-7760/? E//vendor/bin/hw/android.hardware.neuralnetworks@1.1-service-qti: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/listener_android.c:244:listener protocol failure ffffffff
2021-08-19 12:53:32.852 722-7760/? E//vendor/bin/hw/android.hardware.neuralnetworks@1.1-service-qti: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/listener_android.c:251::error: -2147482611: 0 == (nErr = __QAIC_HEADER(adsp_listener_next2)( ctx, nErr, 0, 0, &ctx, &handle, &sc, inBufs, inBufsLen, &inBufsLenReq))
2021-08-19 12:53:32.852 722-7760/? E//vendor/bin/hw/android.hardware.neuralnetworks@1.1-service-qti: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/listener_android.c:333:Error 0x8000040d: listener2 thread exited
2021-08-19 12:53:32.856 722-7761/? E//vendor/bin/hw/android.hardware.neuralnetworks@1.1-service-qti: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/log_config.c:319:Received exit.
2021-08-19 12:53:32.876 722-24072/? E/android.hardware.neuralnetworks@1.1-service: FAILED TO CLOSE LIBRARY
2021-08-19 12:53:32.921 722-7965/? E//vendor/bin/hw/android.hardware.neuralnetworks@1.1-service-qti: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/apps_std_imp.c:729:Error 45: fopen failed for testsig-0xa454e1ab.so. (No such file or directory)
2021-08-19 12:53:32.921 915-964/? E//vendor/bin/cdsprpcd: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/apps_std_imp.c:729:Error 45: fopen failed for testsig-0xa454e1ab.so. (No such file or directory)
2021-08-19 12:53:32.922 722-7965/? E//vendor/bin/hw/android.hardware.neuralnetworks@1.1-service-qti: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/apps_std_imp.c:729:Error 45: fopen failed for testsig.so. (No such file or directory)
2021-08-19 12:53:32.922 915-964/? E//vendor/bin/cdsprpcd: vendor/qcom/proprietary/commonsys-intf/adsprpc/src/apps_std_imp.c:729:Error 45: fopen failed for testsig.so. (No such file or directory)