tensorflow / tensorflow/models

models-r1.13.0 train error

Open
#9,049 0 comments 0 reactions 3 assignees View on GitHub

Nobody has claimed this yet.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Hi, everybody, I am new to tensorflow..
I use the tensorflow-gpu-1.13, cudatoolkit-10.0.130 , cudnn-7.4.1 , and models-r1.13.0.zip. I follow the tutorial , use the frozen inference graph from this Dropbox link https://www.dropbox.com/s/va9ob6wcucusse1/inference_graph.zip?dl=0
I can run the Object_detection_video.py demo successfully....
but then I try to train....

both train.py or model_main.py ,,,,
It can init ok,,,,,, but it always halt after start a while.

#######use model_main.py, halts at step =0.
python model_main.py --pipeline_config_path=training\faster_rcnn_inception_v2_pets.config --model_dir=training --num_train_steps=50000 --sample_1_of_n_eval_examples=1 --alsologtostderr

INFO:tensorflow:loss = 3.8967483, step = 0
I0805 10:27:55.944391 19564 basic_session_run_hooks.py:262] loss = 3.8967483, step = 0

////////////////////////////////////////////////////////////////////////////
#######use train.py, halts at step >=90 , Saving checkpoint
python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config

I0805 10:50:53.543954 18848 learning.py:507] global step 97: loss = 2.5882 (1.244 sec/step)
INFO:tensorflow:global step 98: loss = 2.5304 (1.294 sec/step)
I0805 10:50:54.840963 18848 learning.py:507] global step 98: loss = 2.5304 (1.294 sec/step)
INFO:tensorflow:global step 99: loss = 2.0427 (0.907 sec/step)
I0805 10:50:55.750564 18848 learning.py:507] global step 99: loss = 2.0427 (0.907 sec/step)
INFO:tensorflow:Saving checkpoint to path training/model.ckpt
I0805 10:58:56.402550 2836 supervisor.py:1117] Saving checkpoint to path training/model.ckpt

after halt ,, the anaconda prompt no longer out put any information..
and gpu usage drop to 0%,,, but cpu usage is above 50%, ram 87%

I try to let it run for 10 hours after halt . gpu is always 0%,, cpu is always above 50% ,ram 87%...
but there is no result come out,,,,,
May I should wait for a long time? or it is something errors?
///////////////////////////////////////////////////////////////////////////////////////////////////
for my laptop,
Nitro AN515-52
Core i5-8300H @ 2.30GHz 8 GB RAM
Nvidia GeForce GTX 1050 ( 4 GB )

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.