Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument[[{{node Loss/Where_146}}]]

Prerequisites

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection/legacy/train.py

2. Describe the bug

Training stops suddenly with the following error messages (the same behaviour happened randomly over and over):

`2020-06-25 08:32:54.793317: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at where_op.cc:298 : Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InternalError'>, 2 root error(s) found.
(0) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[node Loss/Where_146 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[gradients/AddN_8/_5341]]
(1) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[node Loss/Where_146 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'Loss/Where_146':
File "object_detection/legacy/train.py", line 185, in
tf.app.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "object_detection/legacy/train.py", line 181, in main
graph_hook_fn=graph_rewriter_fn)
File "/tensorflow/models/research/object_detection/legacy/trainer.py", line 292, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "/tensorflow/models/research/slim/deployment/model_deploy.py", line 194, in create_clones
outputs = model_fn(*args, **kwargs)
File "/tensorflow/models/research/object_detection/legacy/trainer.py", line 207, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 960, in loss
location_losses, cls_losses, prediction_dict, match_list)
File "/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 1227, in _apply_hard_mining
match_list=match_list)
File "/tensorflow/models/research/object_detection/core/losses.py", line 611, in call
self._min_negatives_per_image)
File "/tensorflow/models/research/object_detection/core/losses.py", line 687, in _subsample_selection_to_desired_neg_pos_ratio
tf.logical_or(positives_indicator, topk_negatives_indicator))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 3757, in where
return gen_array_ops.where(condition=condition, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 12326, in where
"Where", input=condition, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

I0625 08:32:54.816522 140568308463424 coordinator.py:224] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InternalError'>, 2 root error(s) found.
(0) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[node Loss/Where_146 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[gradients/AddN_8/_5341]]
(1) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[node Loss/Where_146 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'Loss/Where_146':
File "object_detection/legacy/train.py", line 185, in
tf.app.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "object_detection/legacy/train.py", line 181, in main
graph_hook_fn=graph_rewriter_fn)
File "/tensorflow/models/research/object_detection/legacy/trainer.py", line 292, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "/tensorflow/models/research/slim/deployment/model_deploy.py", line 194, in create_clones
outputs = model_fn(*args, **kwargs)
File "/tensorflow/models/research/object_detection/legacy/trainer.py", line 207, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 960, in loss
location_losses, cls_losses, prediction_dict, match_list)
File "/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 1227, in _apply_hard_mining
match_list=match_list)
File "/tensorflow/models/research/object_detection/core/losses.py", line 611, in call
self._min_negatives_per_image)
File "/tensorflow/models/research/object_detection/core/losses.py", line 687, in _subsample_selection_to_desired_neg_pos_ratio
tf.logical_or(positives_indicator, topk_negatives_indicator))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 3757, in where
return gen_array_ops.where(condition=condition, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 12326, in where
"Where", input=condition, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[{{node Loss/Where_146}}]]
[[gradients/AddN_8/_5341]]
(1) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[{{node Loss/Where_146}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "object_detection/legacy/train.py", line 185, in
tf.app.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "object_detection/legacy/train.py", line 181, in main
graph_hook_fn=graph_rewriter_fn)
File "/tensorflow/models/research/object_detection/legacy/trainer.py", line 417, in train
saver=saver)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 775, in train
train_step_kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 490, in train_step
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[node Loss/Where_146 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[gradients/AddN_8/_5341]]
(1) Internal: WhereOp: Could not launch gpuprim::DeviceReduce::Sum to count number of true / nonzero indices. temp_storage_bytes: 1, status: invalid argument
[[node Loss/Where_146 (defined at /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'Loss/Where_146':
File "object_detection/legacy/train.py", line 185, in
tf.app.run()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "object_detection/legacy/train.py", line 181, in main
graph_hook_fn=graph_rewriter_fn)
File "/tensorflow/models/research/object_detection/legacy/trainer.py", line 292, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "/tensorflow/models/research/slim/deployment/model_deploy.py", line 194, in create_clones
outputs = model_fn(*args, **kwargs)
File "/tensorflow/models/research/object_detection/legacy/trainer.py", line 207, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 960, in loss
location_losses, cls_losses, prediction_dict, match_list)
File "/tensorflow/models/research/object_detection/meta_architectures/ssd_meta_arch.py", line 1227, in _apply_hard_mining
match_list=match_list)
File "/tensorflow/models/research/object_detection/core/losses.py", line 611, in call
self._min_negatives_per_image)
File "/tensorflow/models/research/object_detection/core/losses.py", line 687, in _subsample_selection_to_desired_neg_pos_ratio
tf.logical_or(positives_indicator, topk_negatives_indicator))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py", line 3757, in where
return gen_array_ops.where(condition=condition, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py", line 12326, in where
"Where", input=condition, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
`

3. Steps to reproduce

I fine-tune an ssd+mobilenetv2 using models/research/legacy/train.py . That's all you have to do to reproduce the error.

4. Expected behavior

The behavior is not stable, sometimes it does not occur and I can train for more than 10K steps, sometimes the error happens at an earlier step (600, 8K, 4K...)

6. System information

• OS Platform and Distribution : Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.15.2
• Python version: 3.6
• CUDA/cuDNN version: CUDA 10.0.130 / cuDNN 7.6.5.32-1+cuda10.0_amd64.deb
• GPU model and memory: name: Tesla V100-PCIE-32GB, pci bus id: 0000:3b:00.0, compute capability: 7.0

What could be the problem? the GPU? the tensorflow object detection API code?