tensorflow / tensorflow/models

Add additional channel to pre-trained resnet-50 model

Open
#7,707 2 comments 0 reactions 1 assignee View on GitHub

@jaeyounkim is already working on this.

Since May 15, 2020.

models:research type:community request
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

What is the top-level directory of the model you are using:
from nets.resnet_v1 import bottleneck, resnet_arg_scope

Have I written custom code:
Yes, I am trying to use pre-trained Resnet50 model. However, I intend to modify the first convolution layer to be able to take a 4-channel tensor(image+binary mask), where the network weights (including the first convolutional layer 3 channels) are initialized from the pre-trained model, except for the newly added channel which are initialized randomly.
Adding this feature to existing Resnet models would be really helpful for segmentation tasks.

OS Platform and Distribution:
Ubuntu 16.04

TensorFlow installed from:
Anaconda

TensorFlow version:
1.12.0

Bazel version:
N/A

CUDA/cuDNN version:
CUDA - 10.0

GPU model and memory:
Nvidia Titan Xp 12GB

Exact command to reproduce:
[I intend to add extra channel to the first convolutional layer, while still using the pre-trained Imagenet weights on the other channels of the first convolutional layer]

with tf.contrib.slim.arg_scope(resnet_arg_scope(batch_norm_decay=0.9, weight_decay=0.0)):
with tf.variable_scope('resnet_v1_50', values=[image]) as sc:
end_points_collection = sc.name + '_end_points'
with slim.arg_scope([slim.conv2d, bottleneck],
outputs_collections=end_points_collection):
with slim.arg_scope([slim.batch_norm], is_training=is_training):
net = image
net = conv2d_same(net, 64, 7, stride=2, scope='conv1')
net = slim.max_pool2d(net, [3, 3], stride=2, scope='pool1')

      with tf.variable_scope('block1', values=[net]) as sc_block:
        with tf.variable_scope('unit_1', values=[net]):
          net = bottleneck(net, depth=4*64, depth_bottleneck=64, stride=1)
        with tf.variable_scope('unit_2', values=[net]):
          net = bottleneck(net, depth=4*64, depth_bottleneck=64, stride=1)
        with tf.variable_scope('unit_3', values=[net]):
          net = bottleneck(net, depth=4*64, depth_bottleneck=64, stride=2)

      with tf.variable_scope('block2', values=[net]) as sc_block:
        with tf.variable_scope('unit_1', values=[net]):
          net = bottleneck(net, depth=4*128, depth_bottleneck=128, stride=1)
        with tf.variable_scope('unit_2', values=[net]):
          net = bottleneck(net, depth=4*128, depth_bottleneck=128, stride=1)
        with tf.variable_scope('unit_3', values=[net]):
          net = bottleneck(net, depth=4*128, depth_bottleneck=128, stride=1)
        with tf.variable_scope('unit_4', values=[net]):
          net = bottleneck(net, depth=4*128, depth_bottleneck=128, stride=2)

      with tf.variable_scope('block3', values=[net]) as sc_block:
        with tf.variable_scope('unit_1', values=[net]):
          net = bottleneck(net, depth=4*256, depth_bottleneck=256, stride=1)
        with tf.variable_scope('unit_2', values=[net]):
          net = bottleneck(net, depth=4*256, depth_bottleneck=256, stride=1)
        with tf.variable_scope('unit_3', values=[net]):
          net = bottleneck(net, depth=4*256, depth_bottleneck=256, stride=1)
        with tf.variable_scope('unit_4', values=[net]):
          net = bottleneck(net, depth=4*256, depth_bottleneck=256, stride=1)
        with tf.variable_scope('unit_5', values=[net]):
          net = bottleneck(net, depth=4*256, depth_bottleneck=256, stride=1)
        with tf.variable_scope('unit_6', values=[net]):
          net = bottleneck(net, depth=4*256, depth_bottleneck=256, stride=2)

      net = nonlocal_dot(net, depth=512, embed=True, softmax=True, maxpool=2, scope='nonlocal3')

      with tf.variable_scope('block4', values=[net]) as sc_block:
        with tf.variable_scope('unit_1', values=[net]):
          net = bottleneck(net, depth=4*512, depth_bottleneck=512, stride=1)
        with tf.variable_scope('unit_2', values=[net]):
          net = bottleneck(net, depth=4*512, depth_bottleneck=512, stride=1)
        with tf.variable_scope('unit_3', values=[net]):
          net = bottleneck(net, depth=4*512, depth_bottleneck=512, stride=2)

    net = tf.reduce_mean(net, [1, 2], name='pool5', keep_dims=False)
    endpts = slim.utils.convert_collection_to_dict(
        end_points_collection)
    endpts['model_output'] = endpts['global_pool'] = net

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.