tensorflow / tensorflow/models

Can't train my custom model

Open
#7,912 4 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Jun 19, 2020.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Hi,

I need to train an image recognition with a custom dataset. I found this tutorial online : https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html

I followed carefully all the step and when I launched my training a weird thing happened. It look like the training began but on every image it did not do things even if it takes like 1 minute before adding a new line on the prompt.

"INFO:tensorflow:Starting Session.
I1205 11:38:48.920712 139894946572096 learning.py:754] Starting Session.
INFO:tensorflow:Saving checkpoint to path training/model.ckpt
I1205 11:38:49.326428 139893023946496 supervisor.py:1117] Saving checkpoint to path training/model.ckpt
INFO:tensorflow:Starting Queues.
I1205 11:38:49.371733 139894946572096 learning.py:768] Starting Queues.
INFO:tensorflow:global_step/sec: 0
I1205 11:38:56.664494 139893015553792 supervisor.py:1099] global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
I1205 11:40:51.604840 139893015553792 supervisor.py:1099] global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
I1205 11:42:51.600481 139893015553792 supervisor.py:1099] global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
I1205 11:44:51.600608 139893015553792 supervisor.py:1099] global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
I1205 11:46:51.600104 139893015553792 supervisor.py:1099] global_step/sec: 0
INFO:tensorflow:Saving checkpoint to path training/model.ckpt
I1205 11:48:49.324800 139893023946496 supervisor.py:1117] Saving checkpoint to path training/model.ckpt
INFO:tensorflow:global_step/sec: 0
I1205 11:48:51.599744 139893015553792 supervisor.py:1099] global_step/sec: 0"

Also, when I launch the training I got a lot of warning at the beginning I don't know if it's a part of the problem.

Thank you for your help.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.