tensorflow / tensorflow/models

deeplab - unable to train model_variant=nas_hnasnet

Open
#9,592 0 comments 0 reactions 3 assignees View on GitHub

Nobody has claimed this yet.

models:research type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Hello supporters

I am using tensorflow 1.13.1
with tensorflow model garden version 1.11

2. Describe the bug

I am trying to train for 100 epochs, export a frozen graph , execute it on a few images and check the runtime of each model variant.
i have managed to train with model_variants mobilenet_v2, resnet_v1_50, resnet_v1_50_Beta, resnet_v1_101, resnet_v1_101_beta, xception_41, xception_65, xception_71 , nas_pnasnet using my own dataset.

when i change the model_variant value to nas_hnasnet
train.py fails with the following error

Traceback (most recent call last):
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/train.py", line 628, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/train.py", line 524, in main
dataset.ignore_label)
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/train.py", line 377, in _train_deeplab_model
reuse_variable=(i != 0))
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/train.py", line 275, in _tower_loss
_build_deeplab(iterator, {common.OUTPUT_TYPE: num_of_classes}, ignore_label)
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/train.py", line 235, in _build_deeplab
'total_training_steps': FLAGS.training_number_of_steps,
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/model.py", line 313, in multi_scale_logits
nas_training_hyper_parameters=nas_training_hyper_parameters)
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/model.py", line 553, in _get_logits
nas_training_hyper_parameters=nas_training_hyper_parameters)
File "/home/omerbrandis/ffrobotics/tensorflow-models-master-v1.11/research/deeplab/model.py", line 458, in extract_features
model_options.image_pooling_stride, padding='VALID')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py", line 122, in avg_pool2d
outputs = layer.apply(inputs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 1227, in apply
return self.call(inputs, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/layers/base.py", line 530, in call
outputs = super(Layer, self).call(inputs, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/keras/layers/pooling.py", line 256, in call
data_format=conv_utils.convert_data_format(self.data_format, 4))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 2718, in avg_pool
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 93, in avg_pool
data_format=data_format, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1823, in init
control_input_ops)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1662, in _create_c_op
raise ValueError(str(e))
ValueError: Negative dimension size caused by subtracting 33 from 17 for 'clone_0/AvgPool2D/AvgPool' (op: 'AvgPool') with input shapes: [?,17,17,200].

please advise,
Omer.

--

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.