tensorflow / tensorflow/models

Evaluation couldn't completed after first epoch

Open
#7,996 3 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since May 29, 2020.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Architecture : faster_rcnn_resnet101_coco_2018_01_28
Training Anotated data : 1.3 lakh
Testing anotated data : 40,000
Batch size : 1
num_example : 8000
cuda 10.0 tensorflow 1.14
graphics : GeForce GTX 1080 Ti/PCIe/SSE2
GPU memory : 11 GiB
CPU memory : 15.6 GiB
os : UBUNTU 16.4 LTS

After the first epoch, Evaluation takes too much time and then an error is :

Traceback (most recent call last):
File "/home/dev1/anaconda3/envs/pushti/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/dev1/anaconda3/envs/pushti/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/dev1/anaconda3/envs/pushti/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
(1) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
[[BatchMultiClassNonMaxSuppression/map/TensorArray_11/_3571]]
0 successful operations.
0 derived errors ignored.

How can I fix this bug?
Thank you.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.