DEEPLAB GPU BUG

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [NO ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [ YES] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [YES ] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/deeplab/train.py

2/3. Describe the bug /Steps to reproduce

I wanted to test the deeplab on the ADE20K dataset and force the training on the GPUs by changing
session_config = tf.ConfigProto (allow_soft_placement = True, log_device_placement = False)
to
session_config = tf.ConfigProto (allow_soft_placement = False, log_device_placement = False)
(because when i set allow_soft_placement = True, the training is done on the CPU. ERROR: 2020-07-17 10: 35: 12.537662: W tensorflow / core / common_runtime / colocation_graph.cc: 983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/ job: localhost / replica: 0 / task: 0 / device: CPU: 0])

4. Expected behavior

NA

5. Additional context

python train.py \ --logtostderr \ --training_number_of_steps=150000 \ --train_split="train" \ --model_variant="xception_65" \ --atrous_rates=6 \ --atrous_rates=12 \ --atrous_rates=18 \ --output_stride=16 \ --decoder_output_stride=4 \ --train_crop_size="513,513" \ --train_batch_size=2 \ --min_resize_value=513 \ --max_resize_value=513 \ --resize_factor=16 \ --dataset="ade20k" \ --tf_initial_checkpoint='/hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/backbone/deeplabv3_xception_ade20k_train/model.ckpt' \ --train_logdir='/hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/logs/ade20k/train' \ --dataset_dir='/hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/datasets/ADE20K/tfrecord'

output

`WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/core/conv2d_ws.py:40: The name tf.layers.Layer is deprecated. Please use tf.compat.v1.layers.Layer instead.

WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
• https://github.com/tensorflow/io (for I/O related ops)
  If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From train.py:479: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From train.py:281: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0717 10:39:01.561197 140066995144512 module_wrapper.py:139] From train.py:281: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

WARNING:tensorflow:From train.py:281: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

W0717 10:39:01.561348 140066995144512 module_wrapper.py:139] From train.py:281: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.

WARNING:tensorflow:From train.py:296: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0717 10:39:01.561484 140066995144512 module_wrapper.py:139] From train.py:296: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From train.py:297: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W0717 10:39:01.561623 140066995144512 module_wrapper.py:139] From train.py:297: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:Training on train set
I0717 10:39:01.561691 140066995144512 train.py:297] Training on train set
WARNING:tensorflow:From train.py:322: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0717 10:39:01.562900 140066995144512 module_wrapper.py:139] From train.py:322: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/datasets/data_generator.py:366: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

W0717 10:39:01.567399 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/datasets/data_generator.py:366: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W0717 10:39:01.688875 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

W0717 10:39:01.689791 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.parse_single_example is deprecated. Please use tf.io.parse_single_example instead.

2020-07-17 10:39:02.020326: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-07-17 10:39:02.100824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1b:00.0
2020-07-17 10:39:02.101927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1c:00.0
2020-07-17 10:39:02.102050: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:02.102116: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:02.102189: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:02.102248: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:02.102305: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:02.102361: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:02.106537: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-07-17 10:39:02.106567: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1662] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2
513
513
WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

W0717 10:39:02.457099 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0717 10:39:04.231441 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.lin_space is deprecated. Please use tf.linspace instead.

W0717 10:39:04.232350 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.lin_space is deprecated. Please use tf.linspace instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.

W0717 10:39:04.232610 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.image.resize_bilinear is deprecated. Please use tf.compat.v1.image.resize_bilinear instead.

W0717 10:39:04.408689 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.image.resize_bilinear is deprecated. Please use tf.compat.v1.image.resize_bilinear instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.reverse_v2 is deprecated. Please use tf.reverse instead.

W0717 10:39:06.348407 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.reverse_v2 is deprecated. Please use tf.reverse instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/datasets/data_generator.py:355: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use for ... in dataset: to iterate over a dataset. If using tf.estimator, return the Dataset object directly from your input function. As a last resort, you can use tf.compat.v1.data.make_one_shot_iterator(dataset).
W0717 10:39:06.553466 140066995144512 deprecation.py:323] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/datasets/data_generator.py:355: DatasetV1.make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use for ... in dataset: to iterate over a dataset. If using tf.estimator, return the Dataset object directly from your input function. As a last resort, you can use tf.compat.v1.data.make_one_shot_iterator(dataset).
[Info : start ]: clone device 0
[Info : ok ]: cloned on GPU 0
WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/model.py:320: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

W0717 10:39:06.572277 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/model.py:320: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/core/feature_extractor.py:490: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0717 10:39:06.572718 140066995144512 deprecation.py:323] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/core/feature_extractor.py:490: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/core/xception.py:469: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0717 10:39:06.574600 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/core/xception.py:469: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0717 10:39:06.577203 140066995144512 deprecation.py:323] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/utils/train_utils.py:158: The name tf.losses.add_loss is deprecated. Please use tf.compat.v1.losses.add_loss instead.

W0717 10:39:11.532028 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/utils/train_utils.py:158: The name tf.losses.add_loss is deprecated. Please use tf.compat.v1.losses.add_loss instead.

WARNING:tensorflow:From train.py:336: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

W0717 10:39:11.532345 140066995144512 module_wrapper.py:139] From train.py:336: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From train.py:336: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

W0717 10:39:11.532482 140066995144512 module_wrapper.py:139] From train.py:336: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From train.py:342: The name tf.model_variables is deprecated. Please use tf.compat.v1.model_variables instead.

W0717 10:39:11.533282 140066995144512 module_wrapper.py:139] From train.py:342: The name tf.model_variables is deprecated. Please use tf.compat.v1.model_variables instead.

WARNING:tensorflow:From train.py:343: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

W0717 10:39:11.533433 140066995144512 module_wrapper.py:139] From train.py:343: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From train.py:371: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0717 10:39:12.133874 140066995144512 module_wrapper.py:139] From train.py:371: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

INFO:tensorflow:Setting decay_steps to total training steps.
I0717 10:39:12.136916 140066995144512 train_utils.py:327] Setting decay_steps to total training steps.
WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/utils/train_utils.py:337: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.

W0717 10:39:12.137109 140066995144512 module_wrapper.py:139] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/utils/train_utils.py:337: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.

WARNING:tensorflow:From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/utils/train_utils.py:372: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0717 10:39:12.144202 140066995144512 deprecation.py:323] From /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/utils/train_utils.py:372: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From train.py:390: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

W0717 10:39:12.146084 140066995144512 module_wrapper.py:139] From train.py:390: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

WARNING:tensorflow:From train.py:408: The name tf.check_numerics is deprecated. Please use tf.debugging.check_numerics instead.

W0717 10:39:14.580085 140066995144512 module_wrapper.py:139] From train.py:408: The name tf.check_numerics is deprecated. Please use tf.debugging.check_numerics instead.

WARNING:tensorflow:From train.py:434: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

W0717 10:39:16.576217 140066995144512 module_wrapper.py:139] From train.py:434: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

WARNING:tensorflow:From train.py:437: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

W0717 10:39:16.592415 140066995144512 module_wrapper.py:139] From train.py:437: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

INFO:tensorflow:Ignoring initialization; other checkpoint exists
I0717 10:39:16.595300 140066995144512 train_utils.py:204] Ignoring initialization; other checkpoint exists
WARNING:tensorflow:From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py:742: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
W0717 10:39:18.174265 140066995144512 deprecation.py:323] From /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py:742: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2020-07-17 10:39:19.035738: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2020-07-17 10:39:19.096945: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2020-07-17 10:39:19.101929: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xe07f6b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-17 10:39:19.101999: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-07-17 10:39:19.478954: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xe084600 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-17 10:39:19.479058: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-07-17 10:39:19.479097: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-07-17 10:39:19.485715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1b:00.0
2020-07-17 10:39:19.487777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 1 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:1c:00.0
2020-07-17 10:39:19.488384: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:19.488602: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:19.488774: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:19.488946: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:19.489114: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:19.489282: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2020-07-17 10:39:19.489330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-07-17 10:39:19.489356: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1662] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-07-17 10:39:19.489484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-17 10:39:19.489514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 1
2020-07-17 10:39:19.489538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N Y
2020-07-17 10:39:19.489559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 1: Y N
INFO:tensorflow:Restoring parameters from /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/logs/ade20k/train/model.ckpt-0
I0717 10:39:19.499691 140066995144512 saver.py:1284] Restoring parameters from /hdd-raid0/home_server/houssem/PycharmProjects/models/research/deeplab/logs/ade20k/train/model.ckpt-0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Cannot assign a device for operation image: node image (defined at /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:1 ]. Make sure the device specification refers to a valid device.
[[image]]
I0717 10:39:20.106964 140066995144512 coordinator.py:224] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Cannot assign a device for operation image: node image (defined at /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:1 ]. Make sure the device specification refers to a valid device.
[[image]]
Traceback (most recent call last):
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1348, in _run_fn
self._extend_graph()
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1388, in _extend_graph
tf_session.ExtendSession(self._session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation image: {{node image}} was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:1 ]. Make sure the device specification refers to a valid device.
[[image]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation image: node image (defined at /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:1 ]. Make sure the device specification refers to a valid device.
[[image]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 479, in
tf.app.run()
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "train.py", line 473, in main
save_interval_secs=FLAGS.save_interval_secs)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 753, in train
master, start_standard_services=False, config=session_config) as sess:
File "/usr/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/supervisor.py", line 1014, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/supervisor.py", line 839, in stop
ignore_live_threads=ignore_live_threads)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/supervisor.py", line 1003, in managed_session
start_standard_services=start_standard_services)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/supervisor.py", line 734, in prepare_or_wait_for_session
init_fn=self._init_fn)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/session_manager.py", line 290, in prepare_session
config=config)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/session_manager.py", line 220, in _restore_checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1326, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Cannot assign a device for operation image: node image (defined at /hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:0, /job:localhost/replica:0/task:0/device:XLA_GPU:1 ]. Make sure the device specification refers to a valid device.
[[image]]
ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/absl/app.py", line 321, in run
raise File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv)) File "train.py", line 473, in main
save_interval_secs=FLAGS.save_interval_secs) File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 796, in train
should_retry = True File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/util/tf_should_use.py", line 198, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))

E0717 10:39:20.574415 140066995144512 tf_should_use.py:76] ==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/absl/app.py", line 321, in run
raise File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv)) File "train.py", line 473, in main
save_interval_secs=FLAGS.save_interval_secs) File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/contrib/slim/python/slim/learning.py", line 796, in train
should_retry = True File "/hdd-raid0/home_server/houssem/Envs/deeplab_tf15/lib/python3.6/site-packages/tensorflow_core/python/util/tf_should_use.py", line 198, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))

`

6. System information

• OS Platform : Linux Ubuntu 18.04)
• Mobile device name if the issue happens on a mobile device: NA
• TensorFlow installed from (source or binary): source ( pip )
• TensorFlow / tensorflow-gpu version (use command below): 1.15.3
• Python version: 3.6
• Bazel version (if compiling from source): 2.0
• GCC/Compiler version (if compiling from source): (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
• CUDA/cuDNN version: 10.1/7.6
• GPU model and memory: GeForce GTX 1080 ti driver version 440.82