microsoft / microsoft/onnxruntime-inference-examples
int8 model, how to rescale the predicted bounding boxes
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.7k
- Forks
- 414
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 14
Description
hello,
i had a yolov8 model (.pt) which i wanted to convert to .onnx in int8
since yolo's export function lacks support of onnx int8, i exported it to .tflite with int8 parameter, also using my data.yaml as parameter
then i had a int8.tflite model in saved_model
using tf2onnx.convert i converted .tflite to .onnx (int8)
the problem during Inference is, the bounding boxes need to be rescaled, but i don't know how to find out those x_scale and y_scale parameters
following code works for rescaling, but bounding boxes are not exactly same as the float model,
i manually made out those scaling factors below (i.e. *= 2.65 or *= 3.25) by training many different values, and seeing which seems best.
my question is. how can i find out those values? i resize images to 320x320, original frame size is 2880x1860, pls see following file to see preprocess and postprocess;
https://gist.github.com/hu8813/7699b346e02951dff871083cf6248232
the code i used for Inference:
import cv2
from prepare import YOLOv8
model_path = "./best_int8.onnx"
image_path = "./test.jpg"
frame = cv2.imread(image_path)
detector = YOLOv8(model_path, conf_thres=0.05, iou_thres=0.1)
boxes, scores, class_ids = detector(frame)
for box, score, class_id in zip(boxes, scores, class_ids):
x_min, y_min, x_max, y_max = box
x_min *= 2.65
y_min *= 310
x_max *= 3.25
y_max *= 330
width = (x_max - x_min) / 2
height = (y_max - y_min) / 2
x_center = (x_min + x_max) / 2
y_center = (y_min + y_max) / 2
Here are x_min, y_min, x_max, y_max Values in Float and int8 models:
Float Model:
Frame 1
384.78772 2.7558293 405.63232 2.8917012
413.13806 2.801322 432.97253 2.9235559
356.4293 2.7064931 376.5004 2.8407447
Frame 2
448.05814 2.7975142 509.1641 3.0998619
Int8 Model:
Frame 1
1154.5808 882.2938 1213.4985 925.41516
1241.0344 897.523 1291.4307 936.5822
1068.7329 866.2054 1128.1746 908.5512
Frame 2
1345.1605 893.75183 1526.5665 992.9845
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked preprocessing and postprocessing gist and the prepare.YOLOv8 inference entry point, then compare the float and int8 model outputs for the same frames. Check how the 320x320 resize and the int8 conversion affect bounding-box coordinates; done means the int8 boxes can be mapped back consistently without manually tuned per-coordinate factors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100