Potential bug in mAP computation of Florence-2 fine-tuning notebook
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 9.7k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
Search before asking
- I have searched the Roboflow Notebooks issues and found no similar bug report.
Notebook name
Fine-tuning Florence-2 on Object Detection Dataset
Bug
Predictions from Florence-2 fine-tuned model look like the following:
[Detections(xyxy=array([[ 52.8 , 237.76 , 169.28 , 470.08 ],
[373.44 , 113.6 , 512.32 , 358.08 ],
[161.59999, 330.56 , 301.75998, 585.27997],
[311.36 , 360. , 447.03998, 616.64 ],
[173.12 , 14.4 , 303.03998, 253.12 ]], dtype=float32), mask=None, confidence=array([1., 1., 1., 1., 1.]), class_id=array([34, 50, 46, 2, 33]), tracker_id=None, data={'class_name': array(['9 of hearts', 'queen of hearts', 'king of hearts', '10 of hearts',
'9 of diamonds'], dtype='<U15')}),
Detections(xyxy=array([[3.3056000e+02, 4.2559998e+01, 5.1679999e+02, 2.0703999e+02],
[2.0128000e+02, 8.2239998e+01, 3.8112000e+02, 3.2351999e+02],
[3.1999999e-01, 1.2959999e+02, 2.6719998e+02, 4.1312000e+02],
[1.9808000e+02, 1.7375999e+02, 4.8863998e+02, 4.9887997e+02]],
dtype=float32), mask=None, confidence=array([1., 1., 1., 1.]), class_id=array([16, 24, 32, 28]), tracker_id=None, data={'class_name': array(['5 of clubs', '7 of clubs', '9 of clubs', '8 of clubs'],
dtype='<U10')}),
Detections(xyxy=array([[369.6 , 234.56 , 517.44 , 490.56 ],
[ 87.36 , 163.51999, 255.04 , 402.24 ]], dtype=float32), mask=None, confidence=array([1., 1.]), class_id=array([35, 44]), tracker_id=None, data={'class_name': array(['9 of spades', 'king of clubs'], dtype='<U17')}),
Detections(xyxy=array([[ 56. , 228.79999, 331.84 , 636.48 ]], dtype=float32), mask=None, confidence=array([1.]), class_id=array([31]), tracker_id=None, data={'class_name': array(['8 of spades'], dtype='<U13')})]
It seems that the confidence score is always 1. Wouldn't this cause an issue in creating the precision-recall curve followed by computing mAP?
Environment
NA
Minimal Reproducible Example
NA
Additional
NA
Are you willing to submit a PR?
- Yes I'd like to help by submitting a PR!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with notebooks/how-to-finetune-florence-2-on-detection-dataset.ipynb and inspect the prediction and mAP computation steps. Reproduce the shown Detections output, trace how confidence values feed the precision-recall curve, and verify the metric behavior; done means the confidence handling and mAP result are validated or the affected calculation is clearly identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100