[struct2depth] Mismatch between model definition and provided checkpoint file

System information

• What is the top-level directory of the model you are using: models/research/struct2depth
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): ('v1.14.0-rc1-22-gaf24dc91b5', '1.14.0')
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: Cuda 10.0/cuDNN 7
• GPU model and memory: RTX 2080TI
• Exact command to reproduce: python script.py (where script.py is the script provided below)

Describe the problem

Hi everyone,
I'm attempting to write a streaming wrapper for struct2depth. When I attempt to load the model provided at https://sites.google.com/view/struct2depth, using the code shown below, I get a warning while trying to restore the model:

Shape mismatch, will not restore egomotion_prediction/pose_exp_net/pose/cnv6/weights.
W0704 17:49:22.020884 140487180506944 util.py:206] The following variables in the checkpoint were not loaded:
I0704 17:49:22.020924 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv1/biases
I0704 17:49:22.020999 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv1/weights
I0704 17:49:22.021054 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv2/biases
I0704 17:49:22.021119 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv2/weights
I0704 17:49:22.021184 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv3/biases
I0704 17:49:22.021245 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv3/weights
I0704 17:49:22.021300 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv4/biases
I0704 17:49:22.021347 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv4/weights
I0704 17:49:22.021382 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv5/biases
I0704 17:49:22.021416 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/cnv5/weights
I0704 17:49:22.021452 140487180506944 util.py:208] egomotion_prediction/pose_exp_net/pose/cnv6/weights
I0704 17:49:22.021490 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv1/biases
I0704 17:49:22.021528 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv1/weights
I0704 17:49:22.021563 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv2/biases
I0704 17:49:22.021600 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv2/weights
I0704 17:49:22.021635 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv3/biases
I0704 17:49:22.021668 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv3/weights
I0704 17:49:22.021702 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv4/biases
I0704 17:49:22.021734 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv4/weights
I0704 17:49:22.021768 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv5/biases
I0704 17:49:22.021800 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/cnv5/weights
I0704 17:49:22.021833 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/pose/cnv6/biases
I0704 17:49:22.021867 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/pose/cnv6/weights
I0704 17:49:22.021900 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/pose/cnv7/biases
I0704 17:49:22.021934 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/pose/cnv7/weights
I0704 17:49:22.021966 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/pose/pred/biases
I0704 17:49:22.022000 140487180506944 util.py:208] objectmotion_prediction/pose_exp_net/pose/pred/weights
I0704 17:49:22.022032 140487180506944 util.py:208] train_op/global_step

After this, the code crashes with the error message

tensorflow.python.framework.errors_impl.FailedPreconditionError: 2 root error(s) found.
  (0) Failed precondition: Attempting to use uninitialized value egomotion_prediction/pose_exp_net/pose/cnv6/weights
         [[node egomotion_prediction/pose_exp_net/pose/cnv6/weights/read (defined at depth_est/models/research/struct2depth/nets.py:82) ]]
         [[egomotion_prediction/pose_exp_net/pose/concat/_295]]
  (1) Failed precondition: Attempting to use uninitialized value egomotion_prediction/pose_exp_net/pose/cnv6/weights
         [[node egomotion_prediction/pose_exp_net/pose/cnv6/weights/read (defined at depth_est/models/research/struct2depth/nets.py:82) ]]
0 successful operations.
0 derived errors ignored.

with the respective traceback. Using the provided run_interference.py script (upon which this code is based), this error does not occur, as the supervisor sv will have initialized all variables. However, this leads me to believe, that some part of the graph is default initialized instead of being restored from the checkpoint correctly. Manually loading the checkpoint file, it reveals that the egomotion_prediction/pose_exp_net/pose/cnv6/weights tensor has size (3, 3, 256, 256) instead of (3, 3, 1536, 256) as defined in the model.py file.

Source code / logs

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import sys
import os
from absl import app
from absl import flags
from absl import logging
#import matplotlib.pyplot as plt
import model
import numpy as np
import fnmatch
import tensorflow as tf
import nets
import util

import time

gfile = tf.gfile
def collect_input_images(input_dir, input_list_file, file_extension):
    """Collects all input images that are to be processed."""
    if input_dir is not None:
        im_files = _recursive_glob(input_dir, '*.' + file_extension)
        basepath_in = os.path.normpath(input_dir)
    elif input_list_file is not None:
        im_files = util.read_text_lines(input_list_file)
        basepath_in = os.path.dirname(input_list_file)
        im_files = [os.path.join(basepath_in, f) for f in im_files]
    im_files = [f for f in im_files if 'disp' not in f and '-seg' not in f and
              '-fseg' not in f and '-flip' not in f]
    return sorted(im_files), basepath_in

def _recursive_glob(treeroot, pattern):
    results = []
    for base, _, files in os.walk(treeroot):
        files = fnmatch.filter(files, pattern)
        results.extend(os.path.join(base, f) for f in files)
    return results

class Struct2DepthEst:
    def __init__(self):
        self.sess = tf.Session()
        self.inference_model = model.Model(is_training=False,
                                    batch_size=1,
                                    img_height=128,
                                    img_width=416,
                                    seq_length=3,
                                    architecture=nets.RESNET,
                                    imagenet_norm=True,
                                    use_skip=True,
                                    joint_encoder=True)
        self.model_ckpt = 'model-199160'
        self.input_dir = 'images'
        self.img_width = 416
        self.img_height = 128
        self.batch_size = 1
        self.seq_length = 3
        
        vars_to_restore = util.get_vars_to_save_and_restore(self.model_ckpt)
        self.saver = tf.train.Saver(vars_to_restore)
        self.sv = tf.train.Supervisor(logdir='/tmp/', saver=None)
        self.saver.restore(self.sess, self.model_ckpt)
        self.input_image_seq = []
        
    def run_prediction(self, im):
        im_batch = np.expand_dims(im, axis=0)
        est_depth = self.inference_model.inference_depth(im_batch, self.sess)

        self.input_image_seq.append(im)
        if len(self.input_image_seq) < self.seq_length:  # Buffer not filled yet.
            return
        if len(self.input_image_seq) > self.seq_length:  # Remove oldest entry.
            del self.input_image_seq[0]

        input_image_stack = np.concatenate(self.input_image_seq, axis=2)
        input_image_stack = np.expand_dims(input_image_stack, axis=0)
        est_egomotion = np.squeeze(self.inference_model.inference_egomotion(input_image_stack, self.sess))

if __name__ == '__main__':
    logging.set_verbosity(logging.INFO)           
    input_dir = 'images'
    im_files, basepath_in = collect_input_images(input_dir, None, 'png')
    logging.info('Running inference on %d files.', len(im_files))
    estimator = Struct2DepthEst()
    for i in range(len(im_files)):
        if i % 100 == 0:
            logging.info('%s of %s files processed.', i, len(im_files))
        im = util.load_image(im_files[i], resize=(estimator.img_width, estimator.img_height))
        estimator.run_prediction(im, estimator.sess)