tensorflow / tensorflow/models
Evaluation couldn't completed after first epoch
@pkulzc is already working on this.
Since May 29, 2020.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
Architecture : faster_rcnn_resnet101_coco_2018_01_28
Training Anotated data : 1.3 lakh
Testing anotated data : 40,000
Batch size : 1
num_example : 8000
cuda 10.0 tensorflow 1.14
graphics : GeForce GTX 1080 Ti/PCIe/SSE2
GPU memory : 11 GiB
CPU memory : 15.6 GiB
os : UBUNTU 16.4 LTS
After the first epoch, Evaluation takes too much time and then an error is :
Traceback (most recent call last):
File "/home/dev1/anaconda3/envs/pushti/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/dev1/anaconda3/envs/pushti/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/dev1/anaconda3/envs/pushti/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
(1) Out of range: End of sequence
[[{{node IteratorGetNext}}]]
[[BatchMultiClassNonMaxSuppression/map/TensorArray_11/_3571]]
0 successful operations.
0 derived errors ignored.
How can I fix this bug?
Thank you.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.