DLR-RM / DLR-RM/AugmentedAutoencoder

problem with "tensorflow.python.framework.errors_impl.OutOfRangeError"

Open
#121 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
363
Forks
97
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
HI, I encounter a bug when testing with command "ae_train exp_group/my_mpencoder ", and it always comes out with the same output like this:
"tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence".
I use 8000 jpg images with resolution of 720*540. Also use 3D model in .ply format and convert it into ASCII format with MeshLab software. The model has color. But it finally gets the same problem. Could you help me with that? I attack the whole dialog below. The environment information is also attached.
The terminal information:
jhl@jhl:~/AugmentedAutoencoder_multipath/a_project$ ae_train exp_group/my_mpencoder -d
('zoom_range: ', [0.8, 1.2])
('g_noise: ', False)
('contrast_norm_range: ', [0.5, 2.0])
('mult_brightness: ', [0.6, 1.4])
('max_off_brightness: ', 0.2)
('gaussian_blur: ', True)
('invert: ', True)
WARNING:tensorflow:From /home/jhl/.local/lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:128: DatasetV1.make_initializable_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
This is a deprecated API that should only be used in TF 1 graph mode and legacy TF 2 graph mode available through `tf.compat.v1`. In all other situations -- namely, eager mode and inside `tf.function` -- you can consume dataset elements using `for elem in dataset: ...` or by explicitly creating iterator via `iterator = iter(dataset)` and fetching its elements via `values = next(iterator)`. Furthermore, this API is not available in TF 2. During the transition from TF 1 to TF 2 you can use `tf.compat.v1.data.make_initializable_iterator(dataset)` to create a TF 1 graph mode style iterator for a dataset created through TF 2 APIs. Note that this should be a transient state of your code base as there are in general no guarantees about the interoperability of TF 1 and TF 2 code.
/home/jhl/AugmentedAutoencoder_multipath/a_project/model/bun_zipper2.ply
(None, 128, 128, 3)
0 2
(None, 128, 128, 3)
/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:414: UserWarning: `tf.layers.conv2d` is deprecated and will be removed in a future version. Please Use `tf.keras.layers.Conv2D` instead.
warnings.warn('`tf.layers.conv2d` is deprecated and '
/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1692: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
warnings.warn('`layer.apply` is deprecated and '
/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/normalization.py:308: UserWarning: `tf.layers.batch_normalization` is deprecated and will be removed in a future version. Please use `tf.keras.layers.BatchNormalization` instead. In particular, `tf.control_dependencies(tf.GraphKeys.UPDATE_OPS)` should not be used (consult the `tf.keras.layers.BatchNormalization` documentation).
'`tf.layers.batch_normalization` is deprecated and '
WARNING:tensorflow:From /home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/layers/normalization.py:534: _colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/core.py:329: UserWarning: `tf.layers.flatten` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Flatten` instead.
warnings.warn('`tf.layers.flatten` is deprecated and '
/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/core.py:171: UserWarning: `tf.layers.dense` is deprecated and will be removed in a future version. Please use `tf.keras.layers.Dense` instead.
warnings.warn('`tf.layers.dense` is deprecated and '
1
(128, 128, 3)
[[8, 8], [16, 16], [32, 32], [64, 64]]
(None, 128, 128, 3)
(None, 128, 128, 3)
('tfrecord exists for', '/home/jhl/AugmentedAutoencoder_multipath/a_project/model/bun_zipper2.ply')
1 2 Tensor("RandomShuffle:0", shape=(2,), dtype=int32)
Traceback (most recent call last):
File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
return fn(*args)
File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1360, in _run_fn
target_list, run_metadata)
File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1453, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[{{node my_mpencoder/IteratorGetNext_1}}]]
[[my_mpencoder/IteratorGetNext_1/_7]]
(1) Out of range: End of sequence
[[{{node my_mpencoder/IteratorGetNext_1}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jhl/.local/bin/ae_train", line 8, in
sys.exit(main())
File "/home/jhl/.local/lib/python3.7/site-packages/auto_pose/ae/ae_train.py", line 216, in main
this,_,reconstr_train = sess.run([multi_queue.next_element,multi_queue.next_bg_element,[decoder.x for decoder in decoders]])
File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 968, in run
run_metadata_ptr)
File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1191, in _run
feed_dict_tensor, options, run_metadata)
File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1369, in _do_run
run_metadata)
File "/home/jhl/.local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
(0) Out of range: End of sequence
[[node my_mpencoder/IteratorGetNext_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:174) ]]
[[my_mpencoder/IteratorGetNext_1/_7]]
(1) Out of range: End of sequence
[[node my_mpencoder/IteratorGetNext_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:174) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node my_mpencoder/IteratorGetNext_1:
my_mpencoder/IteratorV2_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:172)

Input Source operations connected to node my_mpencoder/IteratorGetNext_1:
my_mpencoder/IteratorV2_1 (defined at /lib/python3.7/site-packages/auto_pose/ae/multi_queue.py:172)

Original stack trace for 'my_mpencoder/IteratorGetNext_1':
File "/bin/ae_train", line 8, in
sys.exit(main())
File "/lib/python3.7/site-packages/auto_pose/ae/ae_train.py", line 102, in main
iterator = multi_queue.create_iterator(dataset_path, args)
File "/lib/python3.7/site-packages/auto_pose/ae/multi_queue.py", line 174, in create_iterator
self.next_element = iterator.get_next()
File "/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 420, in get_next
name=name)
File "/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2750, in iterator_get_next
output_shapes=output_shapes, name=name)
File "/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 750, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3565, in _create_op_internal
op_def=op_def)
File "/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2045, in __init__
self._traceback = tf_stack.extract_stack_for_node(self._c_op)

**System Info**
Describe the characteristic of your environment:
* GPU models and configuration:NVIDIA GeForce RTX 2060
* Python version:3.7.6
* OpenGL version:3.1.6
* tensorflow-gpu:2.5.0
* CUDA:11.2

**Additional context**
Add any other context about the problem here.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.