Long latency using Yolo tflite model

Hi all,

I exported a generic (and trained) Yolov8n model into tflite format and loaded in a react native app (no expo).

After I have understood the output format, I have been trying to execute a real time inference but I have been facing 2 issues :

• I have a very long latency despite the fact my model weight is only 6mo. The inference time is about 200ms (which is long but explained by the image size of 640 I guess) but what is the weirdest part is that the camera is freezing during much more than that time. For comparison, I have been using also the efficientDet model from the example and it worked fine in real time with very low latency. I actually have no idea what could cause that issue.

• Sorry if this is not completely related to this repo but it might be. My confidence score from the outputs are very very low (0.0000123) and consequently not exploitable. I suspect a wrong input frame during the inference frame which could explain this low score as i'm pretty confident about what I record with my camera. Any insights about what I could possibly do wrong in that case ?

Here is the code :

...
  const model  = useTensorflowModel(require('./android/app/src/main/assets/yolov8n_float16_googleColab.tflite'))//model path
  const actualModel = model.state === 'loaded' ? model.model : undefined

...


const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet'

   

        const resized = resize(frame, {
          scale: {
            width: 640 ,
            height: 640 ,
          },
          pixelFormat: 'rgb',
          dataType: 'float32',
 
        });


        const start = performance.now();

        // Run model with given input buffer synchronously
        const outputs = actualModel.runSync([resized])
        const end = performance.now();
        const executionTime = end - start;
        console.log(`Inference time : ${executionTime.toFixed(2)} ms`);



    const x = valeurs.slice(0,8400)
    const y = valeurs.slice(8400,8400*2)
    const width = valeurs.slice(8400*2,8400*3)
    const height = valeurs.slice(8400*3,8400*4)
    const class1 = valeurs.slice(8400*4,8400*5)
    const class2 = valeurs.slice(8400*5,8400*6)



    const reshaped = []
    for( let i = 0; i < 8400; i++) {
        const detection = []
        detection.push(x[i])
        detection.push(y[i])
        detection.push(width[i])
        detection.push(height[i])
        detection.push(class1[i])
        detection.push(class2[i])
        reshaped.push(detection)
    }

console.log('X', x.slice(0,10))
console.log('Y', y.slice(0,10))
console.log('Width', width.slice(0,10))
console.log('Height', height.slice(0,10))
console.log('Class1', class1.slice(0,10))
console.log('Class2', class2.slice(0,10))

              });

My return jsx :

 <View style={{ width: '100%', aspectRatio: 3/4}}>
  <Camera
    device={device}
    style={StyleSheet.absoluteFill}
    isActive={true}
    frameProcessor={frameProcessor}
    pixelFormat="yuv"
  />
</View>

Thanks for the help!

Start at the frameProcessor using resize(...), actualModel.runSync([resized]), and the Camera configured with pixelFormat="yuv"; compare the reported inference time with the camera freeze. Check the YOLOv8n TFLite input/output handling shown in the snippet, including the 8400-value slicing. Done should identify whether the latency and low confidence come from frame processing or model input/output handling, with a reproducible result.