DeepLab Eval.py "self.__traceback = tf_stack.extract_stack()" error

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/deeplab

2. Describe the bug

When running the eval.py or vis.py code to evaluate my trained model, it outputs a bunch of reports and then says "0 successful operations, 0 derived errors ignored" before saying "self.__traceback = tf_stack.extract_stack()" and stopping.

3. Steps to reproduce

python eval.py \ --logstostderr \ --eval_split="val" \ --model_variant="xception_65" \ --atrous_rates=6 \ --atrous_rates=12 \ --atrous_rates=18 \ --output_stride=16 \ --eval_crop_size=360 \ --eval_crop_size=480

4. Expected behavior

Evaluation/visualization running properly.

5. Additional context

======================End of Report==========================
10 ops no flops stats due to incomplete shapes.
10 ops no flops stats due to incomplete shapes.
INFO:tensorflow:Waiting for new checkpoint at ./Custom_Dataset/Custom_Dataset/Checkpoints
I0717 12:57:58.692667 140108815570752 evaluation.py:189] Waiting for new checkpoint at ./Custom_Dataset/Custom_Dataset/Checkpoints
INFO:tensorflow:Found new checkpoint at ./Custom_Dataset/Custom_Dataset/Checkpoints/model.ckpt-10
I0717 12:57:58.694533 140108815570752 evaluation.py:198] Found new checkpoint at ./Custom_Dataset/Custom_Dataset/Checkpoints/model.ckpt-10
INFO:tensorflow:Graph was finalized.
I0717 12:57:59.233751 140108815570752 monitored_session.py:240] Graph was finalized.
2020-07-17 12:57:59.235746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:3b:00.0
2020-07-17 12:57:59.235861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.236842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 1 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:b1:00.0
2020-07-17 12:57:59.236896: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.237875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 2 with properties:
name: Tesla V100-PCIE-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:d9:00.0
2020-07-17 12:57:59.237915: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-07-17 12:57:59.237927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-07-17 12:57:59.237937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-07-17 12:57:59.237946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-07-17 12:57:59.237956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-07-17 12:57:59.237965: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-07-17 12:57:59.237974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-07-17 12:57:59.239000: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.240044: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.242007: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.243010: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.243974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0, 1, 2
2020-07-17 12:57:59.244085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-17 12:57:59.244095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 1 2
2020-07-17 12:57:59.244101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N Y Y
2020-07-17 12:57:59.244107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 1: Y N Y
2020-07-17 12:57:59.244112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 2: Y Y N
2020-07-17 12:57:59.245311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.246303: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.248255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10845 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-32GB, pci bus id: 0000:3b:00.0, compute capability: 7.0)
2020-07-17 12:57:59.248339: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.249324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 11326 MB memory) -> physical GPU (device: 1, name: Tesla V100-PCIE-32GB, pci bus id: 0000:b1:00.0, compute capability: 7.0)
2020-07-17 12:57:59.249384: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-17 12:57:59.250344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 11299 MB memory) -> physical GPU (device: 2, name: Tesla V100-PCIE-32GB, pci bus id: 0000:d9:00.0, compute capability: 7.0)
INFO:tensorflow:Restoring parameters from ./Custom_Dataset/Custom_Dataset/Checkpoints/model.ckpt-10
I0717 12:57:59.251802 140108815570752 saver.py:1284] Restoring parameters from ./Custom_Dataset/Custom_Dataset/Checkpoints/model.ckpt-10
INFO:tensorflow:Running local_init_op.
I0717 12:58:01.530826 140108815570752 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0717 12:58:01.738378 140108815570752 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Starting evaluation at 2020-07-17-19:58:03
I0717 12:58:03.251836 140108815570752 evaluation.py:450] Starting evaluation at 2020-07-17-19:58:03
2020-07-17 12:58:08.348193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-07-17 12:58:10.332207: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.
2020-07-17 12:58:10.413711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-07-17 12:58:13.579694: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at spacetobatch_op.cc:219 : Invalid argument: padded_shape[0]=58 is not divisible by block_shape[0]=6
2020-07-17 12:58:13.579831: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at spacetobatch_op.cc:219 : Invalid argument: padded_shape[0]=70 is not divisible by block_shape[0]=12
2020-07-17 12:58:13.579901: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at spacetobatch_op.cc:219 : Invalid argument: padded_shape[0]=94 is not divisible by block_shape[0]=18
2020-07-17 12:58:13.724827: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at spacetobatch_op.cc:219 : Invalid argument: padded_shape[0]=58 is not divisible by block_shape[0]=6
2020-07-17 12:58:13.724934: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at spacetobatch_op.cc:219 : Invalid argument: padded_shape[0]=70 is not divisible by block_shape[0]=12
2020-07-17 12:58:13.724978: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at spacetobatch_op.cc:219 : Invalid argument: padded_shape[0]=94 is not divisible by block_shape[0]=18
Traceback (most recent call last):
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:] [x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/Cast_1:0) = ] [2]
[[{{node mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert}}]]
[[ConstantFoldingCtrl/mean_iou/confusion_matrix/assert_less_1/Assert/AssertGuard/Switch_0/_1896]]
(1) Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:] [x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/Cast_1:0) = ] [2]
[[{{node mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "eval.py", line 234, in
tf.app.run()
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "eval.py", line 227, in main
eval_interval_secs=FLAGS.eval_interval_secs)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/contrib/training/python/training/evaluation.py", line 453, in evaluate_repeatedly
session.run(eval_ops, feed_dict)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
run_metadata=run_metadata)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/six.py", line 703, in reraise
raise value
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
run_metadata=run_metadata)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
return self._sess.run(*args, **kwargs)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:] [x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/Cast_1:0) = ] [2]
[[node mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert (defined at /home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[ConstantFoldingCtrl/mean_iou/confusion_matrix/assert_less_1/Assert/AssertGuard/Switch_0/_1896]]
(1) Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:] [x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/Cast_1:0) = ] [2]
[[node mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert (defined at /home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert':
File "eval.py", line 234, in
tf.app.run()
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "eval.py", line 170, in main
weights=weights)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/metrics_impl.py", line 1155, in mean_iou
num_classes, weights)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/metrics_impl.py", line 275, in _streaming_confusion_matrix
labels, predictions, num_classes, weights=weights, dtype=dtypes.float64)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/confusion_matrix.py", line 176, in confusion_matrix
labels, num_classes_int64, message='labels out of bound')],
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/check_ops.py", line 879, in assert_less
summarize, message, name)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/check_ops.py", line 371, in _binary_assert
return control_flow_ops.Assert(condition, data, summarize=summarize)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/util/tf_should_use.py", line 198, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 173, in Assert
guarded_assert = cond(condition, no_op, true_assert, name="AssertGuard")
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1235, in cond
orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 1061, in BuildCondBranch
original_result = fn()
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 171, in true_assert
condition, data, summarize, name="Assert")
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_logging_ops.py", line 74, in _assert
name=name)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/amusaffa/env_3.7.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

Entire log file attached if needed but it is ~2000 lines long so I didn't want to paste it all.
result.txt

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Red Hat 7.8
• Mobile device name if the issue happens on a mobile device: N/A
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): 1.15.0
• Python version: 3.7.7
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 10.0
• GPU model and memory: 3x NVIDIA Tesla V100 (32 gb)