roboflow / roboflow/notebooks

Potential bug in mAP computation of Florence-2 fine-tuning notebook

Open
#294 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Jupyter Notebook
Stars
9.7k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

Search before asking
  • I have searched the Roboflow Notebooks issues and found no similar bug report.
Notebook name

Fine-tuning Florence-2 on Object Detection Dataset

Bug

Predictions from Florence-2 fine-tuned model look like the following:

[Detections(xyxy=array([[ 52.8    , 237.76   , 169.28   , 470.08   ],
        [373.44   , 113.6    , 512.32   , 358.08   ],
        [161.59999, 330.56   , 301.75998, 585.27997],
        [311.36   , 360.     , 447.03998, 616.64   ],
        [173.12   ,  14.4    , 303.03998, 253.12   ]], dtype=float32), mask=None, confidence=array([1., 1., 1., 1., 1.]), class_id=array([34, 50, 46,  2, 33]), tracker_id=None, data={'class_name': array(['9 of hearts', 'queen of hearts', 'king of hearts', '10 of hearts',
        '9 of diamonds'], dtype='<U15')}),
 Detections(xyxy=array([[3.3056000e+02, 4.2559998e+01, 5.1679999e+02, 2.0703999e+02],
        [2.0128000e+02, 8.2239998e+01, 3.8112000e+02, 3.2351999e+02],
        [3.1999999e-01, 1.2959999e+02, 2.6719998e+02, 4.1312000e+02],
        [1.9808000e+02, 1.7375999e+02, 4.8863998e+02, 4.9887997e+02]],
       dtype=float32), mask=None, confidence=array([1., 1., 1., 1.]), class_id=array([16, 24, 32, 28]), tracker_id=None, data={'class_name': array(['5 of clubs', '7 of clubs', '9 of clubs', '8 of clubs'],
       dtype='<U10')}),
 Detections(xyxy=array([[369.6    , 234.56   , 517.44   , 490.56   ],
        [ 87.36   , 163.51999, 255.04   , 402.24   ]], dtype=float32), mask=None, confidence=array([1., 1.]), class_id=array([35, 44]), tracker_id=None, data={'class_name': array(['9 of spades', 'king of clubs'], dtype='<U17')}),
 Detections(xyxy=array([[ 56.     , 228.79999, 331.84   , 636.48   ]], dtype=float32), mask=None, confidence=array([1.]), class_id=array([31]), tracker_id=None, data={'class_name': array(['8 of spades'], dtype='<U13')})]

It seems that the confidence score is always 1. Wouldn't this cause an issue in creating the precision-recall curve followed by computing mAP?

Environment

NA

Minimal Reproducible Example

NA

Additional

NA

Are you willing to submit a PR?
  • Yes I'd like to help by submitting a PR!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with notebooks/how-to-finetune-florence-2-on-detection-dataset.ipynb and inspect the prediction and mAP computation steps. Reproduce the shown Detections output, trace how confidence values feed the precision-recall curve, and verify the metric behavior; done means the confidence handling and mAP result are validated or the affected calculation is clearly identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.