tensorflow / tensorflow/models

MobilenetV2 high vall_loss and false detection

Open
#9,241 5 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Sep 15, 2020.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Hello I am using SSD_Mobilenet V2 from the tensorflow 2 model zoo
I use SSD Mobilenet V2 because I want to make realtime detection using Android Later.
I am using google colab GPU to train
my batch size is 32
and 8000 step

my dataset is form google open image V6 that already anotated
https://github.com/DmitryRyumin/OIDv6

1000 human_body
1000 bicycle
1000 car
1000 motorcycle
336 Stop_sign

I dont understand is this really okay?
image

this is the training progress
I0912 11:40:27.956218 139735407601536 model_lib_v2.py:652] Step 100 per-step time 0.813s loss=1.396
INFO:tensorflow:Step 200 per-step time 0.831s loss=1.192
I0912 11:41:51.976096 139735407601536 model_lib_v2.py:652] Step 200 per-step time 0.831s loss=1.192
INFO:tensorflow:Step 300 per-step time 0.844s loss=1.346
I0912 11:43:14.231966 139735407601536 model_lib_v2.py:652] Step 300 per-step time 0.844s loss=1.346
INFO:tensorflow:Step 400 per-step time 0.771s loss=1.360
I0912 11:44:32.873070 139735407601536 model_lib_v2.py:652] Step 400 per-step time 0.771s loss=1.360
INFO:tensorflow:Step 500 per-step time 0.701s loss=1.444
I0912 11:45:50.882271 139735407601536 model_lib_v2.py:652] Step 500 per-step time 0.701s loss=1.444
INFO:tensorflow:Step 600 per-step time 0.827s loss=1.204
I0912 11:47:12.726692 139735407601536 model_lib_v2.py:652] Step 600 per-step time 0.827s loss=1.204
INFO:tensorflow:Step 700 per-step time 0.736s loss=1.774
I0912 11:48:34.281836 139735407601536 model_lib_v2.py:652] Step 700 per-step time 0.736s loss=1.774
INFO:tensorflow:Step 800 per-step time 0.748s loss=2.155
I0912 11:49:56.143587 139735407601536 model_lib_v2.py:652] Step 800 per-step time 0.748s loss=2.155
INFO:tensorflow:Step 900 per-step time 0.813s loss=2.374
I0912 11:51:17.713723 139735407601536 model_lib_v2.py:652] Step 900 per-step time 0.813s loss=2.374
INFO:tensorflow:Step 1000 per-step time 0.824s loss=2.285
I0912 11:52:35.221740 139735407601536 model_lib_v2.py:652] Step 1000 per-step time 0.824s loss=2.285
INFO:tensorflow:Step 1100 per-step time 0.775s loss=2.309
I0912 11:53:52.562232 139735407601536 model_lib_v2.py:652] Step 1100 per-step time 0.775s loss=2.309
INFO:tensorflow:Step 1200 per-step time 0.757s loss=2.085
I0912 11:55:10.645867 139735407601536 model_lib_v2.py:652] Step 1200 per-step time 0.757s loss=2.085
INFO:tensorflow:Step 1300 per-step time 0.795s loss=2.252
I0912 11:56:29.934082 139735407601536 model_lib_v2.py:652] Step 1300 per-step time 0.795s loss=2.252
INFO:tensorflow:Step 1400 per-step time 0.778s loss=2.198
I0912 11:57:50.456187 139735407601536 model_lib_v2.py:652] Step 1400 per-step time 0.778s loss=2.198
INFO:tensorflow:Step 1500 per-step time 0.912s loss=1.814
I0912 11:59:12.039975 139735407601536 model_lib_v2.py:652] Step 1500 per-step time 0.912s loss=1.814
INFO:tensorflow:Step 1600 per-step time 0.890s loss=2.095
I0912 12:00:32.703614 139735407601536 model_lib_v2.py:652] Step 1600 per-step time 0.890s loss=2.095
INFO:tensorflow:Step 1700 per-step time 0.931s loss=1.791
I0912 12:01:55.470209 139735407601536 model_lib_v2.py:652] Step 1700 per-step time 0.931s loss=1.791
INFO:tensorflow:Step 1800 per-step time 0.832s loss=1.960
I0912 12:03:16.882377 139735407601536 model_lib_v2.py:652] Step 1800 per-step time 0.832s loss=1.960
INFO:tensorflow:Step 1900 per-step time 0.834s loss=1.919
I0912 12:04:38.863093 139735407601536 model_lib_v2.py:652] Step 1900 per-step time 0.834s loss=1.919
INFO:tensorflow:Step 2000 per-step time 0.781s loss=1.836
I0912 12:06:00.716773 139735407601536 model_lib_v2.py:652] Step 2000 per-step time 0.781s loss=1.836
INFO:tensorflow:Step 2100 per-step time 0.784s loss=1.699
I0912 12:07:23.374910 139735407601536 model_lib_v2.py:652] Step 2100 per-step time 0.784s loss=1.699
INFO:tensorflow:Step 2200 per-step time 0.742s loss=1.667
I0912 12:08:45.287693 139735407601536 model_lib_v2.py:652] Step 2200 per-step time 0.742s loss=1.667
INFO:tensorflow:Step 2300 per-step time 0.885s loss=1.775
I0912 12:10:07.288780 139735407601536 model_lib_v2.py:652] Step 2300 per-step time 0.885s loss=1.775
INFO:tensorflow:Step 2400 per-step time 0.802s loss=1.972
I0912 12:11:27.960621 139735407601536 model_lib_v2.py:652] Step 2400 per-step time 0.802s loss=1.972
INFO:tensorflow:Step 2500 per-step time 0.870s loss=1.679
I0912 12:12:52.460410 139735407601536 model_lib_v2.py:652] Step 2500 per-step time 0.870s loss=1.679
INFO:tensorflow:Step 2600 per-step time 0.855s loss=1.824
I0912 12:14:17.548085 139735407601536 model_lib_v2.py:652] Step 2600 per-step time 0.855s loss=1.824
INFO:tensorflow:Step 2700 per-step time 0.850s loss=1.824
I0912 12:15:39.991292 139735407601536 model_lib_v2.py:652] Step 2700 per-step time 0.850s loss=1.824
INFO:tensorflow:Step 2800 per-step time 0.907s loss=1.557
I0912 12:17:01.983814 139735407601536 model_lib_v2.py:652] Step 2800 per-step time 0.907s loss=1.557
INFO:tensorflow:Step 2900 per-step time 0.712s loss=1.685
I0912 12:18:25.104320 139735407601536 model_lib_v2.py:652] Step 2900 per-step time 0.712s loss=1.685
INFO:tensorflow:Step 3000 per-step time 0.896s loss=1.839
I0912 12:19:47.908390 139735407601536 model_lib_v2.py:652] Step 3000 per-step time 0.896s loss=1.839
INFO:tensorflow:Step 3100 per-step time 0.840s loss=1.662
I0912 12:21:11.121691 139735407601536 model_lib_v2.py:652] Step 3100 per-step time 0.840s loss=1.662
INFO:tensorflow:Step 3200 per-step time 0.923s loss=1.586
I0912 12:22:33.857348 139735407601536 model_lib_v2.py:652] Step 3200 per-step time 0.923s loss=1.586
INFO:tensorflow:Step 3300 per-step time 0.739s loss=1.439
I0912 12:23:53.232469 139735407601536 model_lib_v2.py:652] Step 3300 per-step time 0.739s loss=1.439
INFO:tensorflow:Step 3400 per-step time 0.734s loss=1.660
I0912 12:25:11.282731 139735407601536 model_lib_v2.py:652] Step 3400 per-step time 0.734s loss=1.660
INFO:tensorflow:Step 3500 per-step time 0.932s loss=1.530
I0912 12:26:33.676065 139735407601536 model_lib_v2.py:652] Step 3500 per-step time 0.932s loss=1.530
INFO:tensorflow:Step 3600 per-step time 0.880s loss=1.334
I0912 12:27:55.216676 139735407601536 model_lib_v2.py:652] Step 3600 per-step time 0.880s loss=1.334
INFO:tensorflow:Step 3700 per-step time 0.832s loss=1.363
I0912 12:29:19.527122 139735407601536 model_lib_v2.py:652] Step 3700 per-step time 0.832s loss=1.363
INFO:tensorflow:Step 3800 per-step time 0.766s loss=1.343
I0912 12:30:41.062087 139735407601536 model_lib_v2.py:652] Step 3800 per-step time 0.766s loss=1.343
INFO:tensorflow:Step 3900 per-step time 0.764s loss=1.710
I0912 12:32:04.439736 139735407601536 model_lib_v2.py:652] Step 3900 per-step time 0.764s loss=1.710
INFO:tensorflow:Step 4000 per-step time 0.827s loss=1.531
I0912 12:33:28.149232 139735407601536 model_lib_v2.py:652] Step 4000 per-step time 0.827s loss=1.531
INFO:tensorflow:Step 4100 per-step time 0.949s loss=1.545
I0912 12:34:55.035795 139735407601536 model_lib_v2.py:652] Step 4100 per-step time 0.949s loss=1.545
INFO:tensorflow:Step 4200 per-step time 0.912s loss=1.541
I0912 12:36:20.388214 139735407601536 model_lib_v2.py:652] Step 4200 per-step time 0.912s loss=1.541
INFO:tensorflow:Step 4300 per-step time 0.865s loss=1.448
I0912 12:37:44.852299 139735407601536 model_lib_v2.py:652] Step 4300 per-step time 0.865s loss=1.448
INFO:tensorflow:Step 4400 per-step time 0.833s loss=1.298
I0912 12:39:09.155400 139735407601536 model_lib_v2.py:652] Step 4400 per-step time 0.833s loss=1.298
INFO:tensorflow:Step 4500 per-step time 0.827s loss=1.421
I0912 12:40:32.768919 139735407601536 model_lib_v2.py:652] Step 4500 per-step time 0.827s loss=1.421
INFO:tensorflow:Step 4600 per-step time 0.778s loss=1.347
I0912 12:41:53.501947 139735407601536 model_lib_v2.py:652] Step 4600 per-step time 0.778s loss=1.347
INFO:tensorflow:Step 4700 per-step time 0.826s loss=1.466
I0912 12:43:13.038845 139735407601536 model_lib_v2.py:652] Step 4700 per-step time 0.826s loss=1.466
INFO:tensorflow:Step 4800 per-step time 0.755s loss=1.288
I0912 12:44:31.372616 139735407601536 model_lib_v2.py:652] Step 4800 per-step time 0.755s loss=1.288
INFO:tensorflow:Step 4900 per-step time 0.933s loss=1.212
I0912 12:45:51.460758 139735407601536 model_lib_v2.py:652] Step 4900 per-step time 0.933s loss=1.212
INFO:tensorflow:Step 5000 per-step time 0.813s loss=1.464
I0912 12:47:13.726693 139735407601536 model_lib_v2.py:652] Step 5000 per-step time 0.813s loss=1.464
INFO:tensorflow:Step 5100 per-step time 0.854s loss=1.244
I0912 12:48:35.392923 139735407601536 model_lib_v2.py:652] Step 5100 per-step time 0.854s loss=1.244
INFO:tensorflow:Step 5200 per-step time 0.679s loss=1.202
I0912 12:49:53.332885 139735407601536 model_lib_v2.py:652] Step 5200 per-step time 0.679s loss=1.202
INFO:tensorflow:Step 5300 per-step time 0.713s loss=1.580
I0912 12:51:10.413735 139735407601536 model_lib_v2.py:652] Step 5300 per-step time 0.713s loss=1.580
INFO:tensorflow:Step 5400 per-step time 0.812s loss=1.327
I0912 12:52:29.304504 139735407601536 model_lib_v2.py:652] Step 5400 per-step time 0.812s loss=1.327
INFO:tensorflow:Step 5500 per-step time 0.802s loss=1.397
I0912 12:53:49.318393 139735407601536 model_lib_v2.py:652] Step 5500 per-step time 0.802s loss=1.397
INFO:tensorflow:Step 5600 per-step time 0.792s loss=1.503
I0912 12:55:11.936688 139735407601536 model_lib_v2.py:652] Step 5600 per-step time 0.792s loss=1.503
INFO:tensorflow:Step 5700 per-step time 0.933s loss=1.430
I0912 12:56:33.722716 139735407601536 model_lib_v2.py:652] Step 5700 per-step time 0.933s loss=1.430
INFO:tensorflow:Step 5800 per-step time 0.817s loss=1.384
I0912 12:57:55.663592 139735407601536 model_lib_v2.py:652] Step 5800 per-step time 0.817s loss=1.384
INFO:tensorflow:Step 5900 per-step time 0.926s loss=1.433
I0912 12:59:21.656138 139735407601536 model_lib_v2.py:652] Step 5900 per-step time 0.926s loss=1.433
INFO:tensorflow:Step 6000 per-step time 0.886s loss=1.201
I0912 13:00:49.301700 139735407601536 model_lib_v2.py:652] Step 6000 per-step time 0.886s loss=1.201
INFO:tensorflow:Step 6100 per-step time 0.909s loss=1.172
I0912 13:02:14.654530 139735407601536 model_lib_v2.py:652] Step 6100 per-step time 0.909s loss=1.172
INFO:tensorflow:Step 6200 per-step time 0.796s loss=1.339
I0912 13:03:39.308925 139735407601536 model_lib_v2.py:652] Step 6200 per-step time 0.796s loss=1.339
INFO:tensorflow:Step 6300 per-step time 0.882s loss=1.223
I0912 13:05:04.184717 139735407601536 model_lib_v2.py:652] Step 6300 per-step time 0.882s loss=1.223
INFO:tensorflow:Step 6400 per-step time 0.859s loss=1.135
I0912 13:06:30.135948 139735407601536 model_lib_v2.py:652] Step 6400 per-step time 0.859s loss=1.135
INFO:tensorflow:Step 6500 per-step time 0.833s loss=1.351
I0912 13:07:55.786441 139735407601536 model_lib_v2.py:652] Step 6500 per-step time 0.833s loss=1.351
INFO:tensorflow:Step 6600 per-step time 0.925s loss=1.251
I0912 13:09:20.490083 139735407601536 model_lib_v2.py:652] Step 6600 per-step time 0.925s loss=1.251
INFO:tensorflow:Step 6700 per-step time 0.883s loss=1.101
I0912 13:10:44.277643 139735407601536 model_lib_v2.py:652] Step 6700 per-step time 0.883s loss=1.101
INFO:tensorflow:Step 6800 per-step time 0.904s loss=1.522
I0912 13:12:09.175756 139735407601536 model_lib_v2.py:652] Step 6800 per-step time 0.904s loss=1.522
INFO:tensorflow:Step 6900 per-step time 0.797s loss=1.457
I0912 13:13:34.616225 139735407601536 model_lib_v2.py:652] Step 6900 per-step time 0.797s loss=1.457
INFO:tensorflow:Step 7000 per-step time 0.835s loss=1.203
I0912 13:14:57.843070 139735407601536 model_lib_v2.py:652] Step 7000 per-step time 0.835s loss=1.203
INFO:tensorflow:Step 7100 per-step time 0.760s loss=1.240
I0912 13:16:23.126715 139735407601536 model_lib_v2.py:652] Step 7100 per-step time 0.760s loss=1.240
INFO:tensorflow:Step 7200 per-step time 0.822s loss=1.347
I0912 13:17:45.772905 139735407601536 model_lib_v2.py:652] Step 7200 per-step time 0.822s loss=1.347
INFO:tensorflow:Step 7300 per-step time 0.887s loss=1.413
I0912 13:19:11.714976 139735407601536 model_lib_v2.py:652] Step 7300 per-step time 0.887s loss=1.413
INFO:tensorflow:Step 7400 per-step time 0.881s loss=1.129
I0912 13:20:38.058959 139735407601536 model_lib_v2.py:652] Step 7400 per-step time 0.881s loss=1.129
INFO:tensorflow:Step 7500 per-step time 0.906s loss=1.265
I0912 13:22:03.379301 139735407601536 model_lib_v2.py:652] Step 7500 per-step time 0.906s loss=1.265
INFO:tensorflow:Step 7600 per-step time 0.885s loss=1.283
I0912 13:23:29.195028 139735407601536 model_lib_v2.py:652] Step 7600 per-step time 0.885s loss=1.283
INFO:tensorflow:Step 7700 per-step time 0.807s loss=1.330
I0912 13:24:54.521943 139735407601536 model_lib_v2.py:652] Step 7700 per-step time 0.807s loss=1.330
INFO:tensorflow:Step 7800 per-step time 0.784s loss=1.258
I0912 13:26:17.378603 139735407601536 model_lib_v2.py:652] Step 7800 per-step time 0.784s loss=1.258
INFO:tensorflow:Step 7900 per-step time 0.866s loss=1.098
I0912 13:27:42.443229 139735407601536 model_lib_v2.py:652] Step 7900 per-step time 0.866s loss=1.098
INFO:tensorflow:Step 8000 per-step time 0.893s loss=1.235
I0912 13:29:03.420507 139735407601536 model_lib_v2.py:652] Step 8000 per-step time 0.893s loss=1.235

and I give 3 photo of a human to test it
the first test is myself formal photo and it detect as motorcycle 58%
the second is photo of my friend look formal and it cannot detect anything
the third photo is 5 human taking photo and cannot detect anything

Can anybody help me with this?
or give me some advice
is this my dataset problem?
batch size problem?
or need more steps?

this is my learning rate
image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.