Frame Resizing and Drawing Confusion

No file or test is named. Start from the reported resize configuration and the frame dimensions used before drawing, then compare the resize plugin's cropping behavior with EfficientDet's normalized box coordinates on Android. Done means the coordinate mapping is reproducible and the rendered boxes align with the camera preview, or the required correction is clearly documented.

Not sure if this is just me or a complexity that could do with abstracting out.

I've set up on an android.

My camera is rendering square on screen, and seems to be showing dead center of what the lens shows.

The frame that the frameProcessor is receiving is not a 1/1 aspect ratio as there doesn't seem to be an option to do that. It often receives 1080 x800 or some similar ratio.

This is then resized to 320x320 by the resize plugin, which if I understand correctly, resizes and crops to the center by default.

I'm using effecientdet which seems to return normalized bbox coordinates. My issue is converting those normalized values back to the full frame for rendering the bbox to the screen. I'm taking the normalized value and multiplying by framewidth and frameheight but of course these don't map exactly back to the original aspect ratio. So the box is rendered in the wrong place (obviously).

I'm wondering if I'm missing something obvious here.

`const resized = resize(frame, {
scale: {
width: 320,
height: 320,
},

    pixelFormat: 'rgb',
    dataType: 'uint8',
  })
  const result = actualModel.runSync([resized])

  // const num_detections = result[3]?.[0] ?? 0
  // console.log('Number of Detections: ' + num_detections)

  const detection_scores = result[2]
  const detection_boxes = result[0]

  frame.render();
  const frameWidth = frame.width;
  const frameHeight = frame.height;

  
  let thisDetectedClasses: string[] = []

  for (let i = 0; i < detection_boxes.length; i += 4) {
    const confidence = detection_scores[i / 4]
    if (confidence > 0.6) {
      
      const classId = result[1][i / 4] || 0
      const className = cocoLabels[Number(classId)] || `Class ${classId}`;

      thisDetectedClasses.push(className)
      // console.log(`Detected ${className} with confidence: ${confidence}`)

      const left = frameWidth * detection_boxes[i]
      const top = frameHeight * detection_boxes[i + 1]
      const right = frameWidth * detection_boxes[i + 2]
      const bottom = frameHeight * detection_boxes[i + 3]
      

      const rect = Skia.XYWHRect(left, top, 10, 10);
      const paint = Skia.Paint();
      paint.setColor(Skia.Color('red'));
      paint.setStyle(PaintStyle.Stroke);
      paint.setStrokeWidth(4);
      frame.drawRect(rect, paint);
    }
  }`