Excessive Redundant Bounding Boxes in PaliGemma2 Fine-Tuning for Detection Tasks
Open
Nobody has claimed this yet.
bug
- Dominant language
- Jupyter Notebook
- Stars
- 9.7k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
Search before asking
- I have searched the Roboflow Notebooks issues and found no similar bug report.
Notebook name
Bug
Thank you for your excellent work. While attempting to fine-tune for a downstream detection task based on PaliGemma2, I noticed that the final training results produced many redundant bounding boxes. Based on the predictions, the model is capable of detecting the targets, but it continuously outputs additional bounding boxes until it reaches the set max-new-tokens limit. Could you provide any insights or suggestions on this issue?
<loc0400><loc0516><loc0652><loc0712> 7 of clubs ; <loc0292><loc0300><loc0584><loc0512> 8 of clubs ; <loc0406><loc0724><loc0708><loc1007> 5 of clubs ; <loc0216><loc0084><loc0528><loc0316> 6 of clubs ; <loc0400><loc0516><loc0648><loc0708> 6 of clubs ; <loc0292><loc0295><loc0580><loc0512> 8 of clubs ; <loc0412><loc0732><loc0701><loc1007> 4 of clubs ; <loc0208><loc0080><loc0528><loc0316> 5 of clubs ; <loc0756><loc0136><loc1023><loc0316> 10 of clubs ; <loc0000><loc0000><loc1023><loc1016> 9 of clubs ; <loc0000><loc0000><loc0580><loc0540> 10 of clubs ; <loc0416><loc0540><loc0644><loc0708> 8 of clubs ; <loc0756><loc0144><loc0880><loc0292> 5 of clubs ; <loc0756><loc0144><loc1023><loc0322> 2 of clubs ; <loc0756><loc0144><loc1023><loc0316> 9 of clubs ; <loc0756><loc0144><loc1023><loc0316> 5 of clubs ; <loc0756><loc0144><loc1023><loc0305> 5 of clubs ; <loc0756><loc0144><loc1023><loc0305> 5 of clubs ; <loc0756><loc0144><loc1023><loc0295> 5 of clubs ; <loc0756><loc0232><loc1023><loc0322> 5 of clubs ; <loc0738><loc0000><loc1023><loc0136> 5 of clubs ; <loc0756><loc0000><loc1023><loc0136> 5 of clubs ; <loc0756><loc0000><loc1023><loc0136> 5 of clubs ; <loc0000><loc0000><loc0580><loc0372> 10 of clubs ; <loc0738><loc0000><loc1023><loc0136> 5 of clubs ; <loc0738><loc0000><loc1023>
Environment
- Local
- OS: Ubuntu 20.04
- Python: 3.10.6
- Transformers: 4.47.0
Minimal Reproducible Example
No response
Additional
Additionally, here is the terminal log output:
ubuntu@ubuntu:/ssd2/workspace/mllm/fine-tune-paligemma/Google-PaliGemma2-Finetune$ CUDA_VISIBLE_DEVICES=2,3 python train.py --lora --epochs 8
hyperparameters: remove_unused_columns=False, gradient_accumulation_steps=16, warmup_steps=2, weight_decay=1e-06, adam_beta2=0.999, logging_steps=50, optim=adamw_hf, save_strategy=steps, save_steps=200, save_total_limit=1, bf16=True, report_to=['tensorboard'], dataloader_pin_memory=False
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00, 3.00s/it]
trainable params: 11,876,352 || all params: 3,045,003,504 || trainable%: 0.3900
freezing vision model layers
freezing multi-modal projector
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
{'loss': 2.6005, 'grad_norm': 23.956716537475586, 'learning_rate': 1.7587939698492464e-05, 'epoch': 0.99}
{'loss': 1.9132, 'grad_norm': 23.055450439453125, 'learning_rate': 1.5075376884422112e-05, 'epoch': 1.97}
{'loss': 1.6768, 'grad_norm': 33.97260284423828, 'learning_rate': 1.256281407035176e-05, 'epoch': 2.95}
{'loss': 1.5855, 'grad_norm': 26.143875122070312, 'learning_rate': 1.0050251256281408e-05, 'epoch': 3.93}
{'loss': 1.5406, 'grad_norm': 24.072601318359375, 'learning_rate': 7.537688442211056e-06, 'epoch': 4.91}
{'loss': 1.515, 'grad_norm': 34.959720611572266, 'learning_rate': 5.025125628140704e-06, 'epoch': 5.89}
{'loss': 1.5009, 'grad_norm': 29.38210105895996, 'learning_rate': 2.512562814070352e-06, 'epoch': 6.87}
{'loss': 1.4799, 'grad_norm': 39.997161865234375, 'learning_rate': 0.0, 'epoch': 7.85}
{'train_runtime': 5094.8047, 'train_samples_per_second': 1.273, 'train_steps_per_second': 0.079, 'train_loss': 1.726562728881836, 'epoch': 7.85}
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [1:24:54<00:00, 12.74s/it]
0%| | 0/44 [00:00<?, ?it/s]The 'batch_size' attribute of HybridCache is deprecated and will be removed in v4.49. Use the more precisely named 'self.max_batch_size' attribute instead.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 44/44 [32:50<00:00, 44.77s/it]
map_result: MeanAveragePrecisionResult:
Metric target: MetricTarget.BOXES
Class agnostic: False
mAP @ 50:95: 0.4415
mAP @ 50: 0.4892
mAP @ 75: 0.4738
mAP scores: [0.48919323 0.48853667 0.48907614 0.47919466 0.47776663 0.47379989
0.47379989 0.4553695 0.38925978 0.19944912]
IoU thresh: [0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]
AP per class:
0: [0.03280268 0.03280268 0.03280268 0.03280268 0.03280268 0.03280268
0.03280268 0.03280268 0.03280268 0.02328249]
1: [0.07497781 0.07497781 0.07497781 0.07497781 0.07497781 0.07497781
0.07497781 0.07497781 0.07497781 0.02398884]
2: [0.05993939 0.05993939 0.05993939 0.05993939 0.05993939 0.05993939
0.05993939 0.05993939 0.04830372 0.03577281]
3: [0.07545013 0.07545013 0.07545013 0.07545013 0.07545013 0.07545013
0.07545013 0.07545013 0.07545013 0.04084158]
4: [0.23377338 0.23377338 0.23377338 0.23377338 0.23377338 0.23377338
0.23377338 0.23377338 0.23377338 0.12376238]
5: [0.1980198 0.1980198 0.1980198 0.1980198 0.1980198 0.1980198 0.1980198
0.1980198 0.1980198 0.1980198]
6: [0.24752475 0.24752475 0.24752475 0.24752475 0.24752475 0.24752475
0.24752475 0.24752475 0.24752475 0.24752475]
7: [0.23883888 0.23883888 0.23883888 0.23883888 0.23883888 0.23883888
0.23883888 0.09207921 0.09207921 0. ]
8: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
0.330033 0.330033 0.330033 0.08250825]
9: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
0.330033 0.330033]
10: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
0.4950495 0.4950495 0.4950495]
11: [0.32850071 0.32850071 0.32850071 0.32850071 0.32850071 0.32850071
0.32850071 0.32850071 0.12835926 0.04084158]
12: [0.7029703 0.7029703 0.7029703 0.7029703 0.7029703 0.7029703 0.7029703
0.7029703 0.7029703 0. ]
13: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
14: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
0.4950495 0.4950495 0. ]
15: [0.32850071 0.32850071 0.32850071 0.32850071 0.32850071 0.32850071
0.32850071 0.32850071 0.20226308 0.0466761 ]
16: [0.46153408 0.42739274 0.34818482 0.34818482 0.27392739 0.27392739
0.27392739 0.27392739 0.27392739 0.27392739]
17: [0.34771334 0.34771334 0.34771334 0.34771334 0.34771334 0.34771334
0.34771334 0.34771334 0.34771334 0.16184311]
18: [0.34470678 0.34470678 0.34470678 0.34470678 0.34470678 0.34470678
0.34470678 0.34470678 0.19455264 0.04158416]
19: [0.11261932 0.11261932 0.11261932 0.11261932 0.11261932 0.11261932
0.11261932 0.07263803 0.0090009 0. ]
20: [0.76661952 0.76661952 0.76661952 0.76661952 0.76661952 0.76661952
0.76661952 0.76661952 0.47100424 0.04950495]
21: [0.62164074 0.62164074 0.62164074 0.62164074 0.62164074 0.62164074
0.62164074 0.62164074 0.62164074 0.04950495]
22: [0.83168317 0.83168317 0.83168317 0.83168317 0.83168317 0.83168317
0.83168317 0.83168317 0.61110325 0. ]
23: [0.71047105 0.71047105 0.71047105 0.47794779 0.47794779 0.47794779
0.47794779 0.47794779 0.08550855 0.04950495]
24: [0.51815182 0.51815182 0.51815182 0.51815182 0.51815182 0.51815182
0.51815182 0.51815182 0.51815182 0.43894389]
25: [0.71239981 0.71239981 0.71239981 0.71239981 0.71239981 0.71239981
0.71239981 0.71239981 0.71239981 0.5709571 ]
26: [0.6165732 0.6165732 0.6165732 0.6165732 0.6165732 0.6165732
0.6165732 0.6165732 0.52602183 0.20660066]
27: [0.72811567 0.72811567 0.72811567 0.44680182 0.44680182 0.44680182
0.44680182 0.15558699 0.07260726 0. ]
28: [0.74422442 0.74422442 0.74422442 0.74422442 0.74422442 0.5379538
0.5379538 0.5379538 0.5379538 0.34818482]
29: [0.77310231 0.77310231 0.77310231 0.77310231 0.77310231 0.77310231
0.77310231 0.77310231 0.77310231 0.14438944]
30: [0.74014555 0.74014555 0.74014555 0.74014555 0.74014555 0.74014555
0.74014555 0.74014555 0.74014555 0.17161716]
31: [0.55941981 0.55941981 0.55941981 0.55941981 0.55941981 0.55941981
0.55941981 0.55941981 0.55941981 0.01414427]
32: [0.47854785 0.47854785 0.47854785 0.47854785 0.47854785 0.47854785
0.47854785 0.47854785 0.47854785 0.22277228]
33: [0.42285479 0.42285479 0.42285479 0.42285479 0.42285479 0.42285479
0.42285479 0.42285479 0.24752475 0.25636492]
34: [0.2491377 0.2491377 0.2491377 0.2491377 0.2491377 0.2491377
0.2491377 0.10245912 0.10245912 0.04479019]
35: [0.08392268 0.08392268 0.08392268 0.08392268 0.08392268 0.08392268
0.08392268 0.08392268 0.08392268 0.01815182]
36: [0.44554455 0.44554455 0.44554455 0.44554455 0.44554455 0.44554455
0.44554455 0.44554455 0.44554455 0.0990099 ]
37: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
0.4950495 0.4950495 1. ]
38: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
39: [0.44059406 0.44059406 0.44059406 0.44059406 0.44059406 0.44059406
0.44059406 0.44059406 0.4950495 0.08168317]
40: [1. 1. 1. 1. 1. 1.
1. 1. 1. 0.69059406]
41: [0.17030549 0.17030549 0.17030549 0.17030549 0.17030549 0.17030549
0.17030549 0.17030549 0.13645595 0.00634679]
42: [0.8019802 0.8019802 0.8019802 0.8019802 0.8019802 0.8019802 0.8019802
0.8019802 0.8019802 0.0990099]
43: [0.91584158 0.91584158 0.80693069 0.80693069 0.80693069 0.80693069
0.80693069 0.80693069 0.08168317 0. ]
44: [0.81848185 0.81848185 0.81848185 0.81848185 0.81848185 0.81848185
0.81848185 0.81848185 0.81848185 0.1320132 ]
45: [0.25990099 0.25990099 0.47607261 0.47607261 0.47607261 0.47607261
0.47607261 0.14232673 0.14232673 0.14232673]
46: [0.82791136 0.82791136 0.82791136 0.82791136 0.82791136 0.82791136
0.82791136 0.82791136 0.82791136 0.42479962]
47: [0.7019802 0.7019802 0.7019802 0.7019802 0.7019802 0.7019802
0.7019802 0.7019802 0.06534653 0. ]
48: [0.09806695 0.09806695 0.09806695 0.09806695 0.09806695 0.09806695
0.09806695 0.09806695 0.09806695 0.00884017]
49: [0.62871287 0.62871287 0.62871287 0.62871287 0.62871287 0.62871287
0.62871287 0.62871287 0.62871287 0.62871287]
50: [0.71287129 0.71287129 0.71287129 0.71287129 0.71287129 0.71287129
0.71287129 0.71287129 0.42574257 0.30693069]
51: [0.12575994 0.12575994 0.12575994 0.12575994 0.12575994 0.12575994
0.12575994 0.12575994 0.12575994 0. ]
Small objects:
MeanAveragePrecisionResult:
Metric target: MetricTarget.BOXES
Class agnostic: False
mAP @ 50:95: 0.0000
mAP @ 50: 0.0000
mAP @ 75: 0.0000
mAP scores: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
IoU thresh: [0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]
AP per class:
No results
Medium objects:
MeanAveragePrecisionResult:
Metric target: MetricTarget.BOXES
Class agnostic: False
mAP @ 50:95: 0.0000
mAP @ 50: 0.0000
mAP @ 75: 0.0000
mAP scores: [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
IoU thresh: [0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]
AP per class:
No results
Large objects:
MeanAveragePrecisionResult:
Metric target: MetricTarget.BOXES
Class agnostic: False
mAP @ 50:95: 0.4800
mAP @ 50: 0.5283
mAP @ 75: 0.5193
mAP scores: [0.52829363 0.52765895 0.53252578 0.52467807 0.52340872 0.51931504
0.51931504 0.49847883 0.42658422 0.20021746]
IoU thresh: [0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95]
AP per class:
0: [0.03619003 0.03619003 0.03619003 0.03619003 0.03619003 0.03619003
0.03619003 0.03619003 0.03619003 0.02657742]
1: [0.05798595 0.05798595 0.05798595 0.05798595 0.05798595 0.05798595
0.05798595 0.05798595 0.05798595 0.0256342 ]
2: [0.04018435 0.04018435 0.04018435 0.04018435 0.04018435 0.04018435
0.04018435 0.04018435 0.0289021 0.01678454]
3: [0.11543189 0.11543189 0.11543189 0.11543189 0.11543189 0.11543189
0.11543189 0.11543189 0.11543189 0.08168317]
4: [0.26520509 0.26520509 0.26520509 0.26520509 0.26520509 0.26520509
0.26520509 0.26520509 0.26520509 0.12376238]
5: [0.24752475 0.24752475 0.24752475 0.24752475 0.24752475 0.24752475
0.24752475 0.24752475 0.24752475 0.24752475]
6: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
0.330033 0.330033]
7: [0.56831683 0.56831683 0.56831683 0.56831683 0.56831683 0.56831683
0.56831683 0.4019802 0.4019802 0. ]
8: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
0.330033 0.330033 0.330033 0.08250825]
9: [0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033 0.330033
0.330033 0.330033]
10: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
0.4950495 0.4950495 0.4950495]
11: [0.62623762 0.62623762 0.62623762 0.62623762 0.62623762 0.62623762
0.62623762 0.62623762 0.41831683 0.33663366]
12: [0.75247525 0.75247525 0.75247525 0.75247525 0.75247525 0.75247525
0.75247525 0.75247525 0.75247525 0. ]
13: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
14: [0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495 0.4950495
0.4950495 0.4950495 0. ]
15: [0.57953795 0.57953795 0.57953795 0.57953795 0.57953795 0.57953795
0.57953795 0.57953795 0.1379538 0.03630363]
16: [0.44494449 0.41194119 0.33993399 0.33993399 0.27392739 0.27392739
0.27392739 0.27392739 0.27392739 0.27392739]
17: [0.37600189 0.37600189 0.37600189 0.37600189 0.37600189 0.37600189
0.37600189 0.37600189 0.37600189 0.16184311]
18: [0.33200051 0.33200051 0.33200051 0.33200051 0.33200051 0.33200051
0.33200051 0.33200051 0.18223141 0.03272827]
19: [0.34899919 0.34899919 0.34899919 0.34899919 0.34899919 0.34899919
0.34899919 0.19309074 0.0330033 0. ]
20: [0.81188119 0.81188119 0.81188119 0.81188119 0.81188119 0.81188119
0.81188119 0.81188119 0.5049505 0.04950495]
21: [0.48797737 0.48797737 0.48797737 0.48797737 0.48797737 0.48797737
0.48797737 0.48797737 0.48797737 0.04950495]
22: [0.74257426 0.74257426 0.74257426 0.74257426 0.74257426 0.74257426
0.74257426 0.74257426 0.5675389 0. ]
23: [0.60003143 0.60003143 0.60003143 0.39336791 0.39336791 0.39336791
0.39336791 0.39336791 0.07150715 0.04950495]
24: [0.57001414 0.57001414 0.57001414 0.57001414 0.57001414 0.57001414
0.57001414 0.57001414 0.57001414 0.49080622]
25: [0.71239981 0.71239981 0.71239981 0.71239981 0.71239981 0.71239981
0.71239981 0.71239981 0.71239981 0.5709571 ]
26: [0.60591059 0.60591059 0.60591059 0.60591059 0.60591059 0.60591059
0.60591059 0.60591059 0.51749175 0.20660066]
27: [0.59619491 0.59619491 0.59619491 0.39477771 0.39477771 0.39477771
0.39477771 0.11745292 0.08958039 0. ]
28: [0.77062706 0.77062706 0.77062706 0.77062706 0.77062706 0.55775578
0.55775578 0.55775578 0.55775578 0.36138614]
29: [0.77310231 0.77310231 0.77310231 0.77310231 0.77310231 0.77310231
0.77310231 0.77310231 0.77310231 0.14438944]
30: [0.73443344 0.73443344 0.73443344 0.73443344 0.73443344 0.73443344
0.73443344 0.73443344 0.73443344 0.16984006]
31: [0.6039604 0.6039604 0.6039604 0.6039604 0.6039604 0.6039604 0.6039604
0.6039604 0.6039604 0.0330033]
32: [0.51980198 0.51980198 0.51980198 0.51980198 0.51980198 0.51980198
0.51980198 0.51980198 0.51980198 0.24339934]
33: [0.41136256 0.41136256 0.41136256 0.41136256 0.41136256 0.41136256
0.41136256 0.41136256 0.25636492 0.24752475]
34: [0.26440296 0.26440296 0.26440296 0.26440296 0.26440296 0.26440296
0.26440296 0.11423612 0.11423612 0.04696624]
35: [0.15470297 0.15470297 0.15470297 0.15470297 0.15470297 0.15470297
0.15470297 0.15470297 0.15470297 0.04084158]
36: [0.41254125 0.41254125 0.41254125 0.41254125 0.41254125 0.41254125
0.41254125 0.41254125 0.41254125 0.12376238]
37: [1. 1. 1. 1. 1. 1. 1.
1. 1. 0.4950495]
38: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
39: [0.7019802 0.7019802 0.7019802 0.7019802 0.7019802 0.7019802
0.7019802 0.7019802 0.75643564 0.06534653]
40: [1. 1. 1. 1. 1. 1.
1. 1. 1. 0.69059406]
41: [0.17030549 0.17030549 0.17030549 0.17030549 0.17030549 0.17030549
0.17030549 0.17030549 0.13645595 0.00634679]
42: [0.75247525 0.75247525 0.75247525 0.75247525 0.75247525 0.75247525
0.75247525 0.75247525 0.75247525 0.0990099 ]
43: [0.80693069 0.80693069 0.91584158 0.91584158 0.91584158 0.91584158
0.91584158 0.91584158 0.08168317 0. ]
44: [0.81848185 0.81848185 0.81848185 0.81848185 0.81848185 0.81848185
0.81848185 0.81848185 0.81848185 0.1320132 ]
45: [0.25990099 0.25990099 0.47607261 0.47607261 0.47607261 0.47607261
0.47607261 0.14232673 0.14232673 0.14232673]
46: [0.87741631 0.87741631 0.87741631 0.87741631 0.87741631 0.87741631
0.87741631 0.87741631 0.87741631 0.39179632]
47: [0.75643564 0.75643564 0.75643564 0.75643564 0.75643564 0.75643564
0.75643564 0.75643564 0.06534653 0. ]
48: [0.09806695 0.09806695 0.09806695 0.09806695 0.09806695 0.09806695
0.09806695 0.09806695 0.09806695 0.00884017]
49: [0.61103253 0.61103253 0.61103253 0.61103253 0.61103253 0.61103253
0.61103253 0.61103253 0.61103253 0.61103253]
50: [0.64686469 0.64686469 0.64686469 0.64686469 0.64686469 0.64686469
0.64686469 0.64686469 0.45874587 0.33993399]
51: [0.42822549 0.42822549 0.42822549 0.42822549 0.42822549 0.42822549
0.42822549 0.42822549 0.42822549 0. ]
Are you willing to submit a PR?
- Yes I'd like to help by submitting a PR!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with notebooks/how-to-finetune-paligemma2-on-detection-dataset.ipynb and the reported train.py command, then review how training output is generated and limited by max-new-tokens. Reproduce the redundant-box output if possible and compare it with the reported Transformers 4.47.0 environment. Done means identifying a supported cause and documenting actionable guidance or a confirmed correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100