LSTM Object Detection Model Does Not Run

System information

• What is the top-level directory of the model you are using: lstm_object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Trying to
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): 1.12.0
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 9.0
• GPU model and memory: GTX 1070 ti
• Exact command to reproduce: python train.py --train_dir=training --pipeline_config_path=configs/lstm_ssd_mobilenet_v1_imagenet.config

Describe the problem

Training the LSTM object detection model does not work. After making a tfrecord, modifying the config as necessary, creating a training dir, and running the command, I get this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Tried to explicitly squeeze dimension 0 but dimension was not 1: 0
         [[Node: Squeeze_1 = Squeeze[T=DT_INT64, squeeze_dims=[0], _device="/job:localhost/replica:0/task:0/device:CPU:0"](split_2)]]

More documentation, including a simple example file of how to make a tfrecord and train, would be very helpful. I have tried two ways to create a tfrecord, both of which are shown below. I thought maybe the record structure is wrong, but if I put a typo in the record keys, I get a different error complaining about that, so perhaps I structured the records correctly. I tried looking at the model in tensorboard and modifying the training code in slim/learning.py to fetch values from individual nodes near Squeeze_1. I print the node, the output, and the shape of the output. Here are results from these attempts:

try run:  split_1:0
Tensor("split_1:0", shape=(?, ?, 4), dtype=float32, device=/device:CPU:0)
value: []
size: (0, 0, 4)

try run:  ParseSingleSequenceExample/ParseSingleSequenceExample:0
Tensor("ParseSingleSequenceExample/ParseSingleSequenceExample:0", shape=(), dtype=string, device=/device:CPU:0)
value: b''
size: ()

try run:  ResizeImage/resize_images/ResizeBilinear:0
Tensor("ResizeImage/resize_images/ResizeBilinear:0", shape=(4, 256, 256, 3), dtype=float32, device=/device:CPU:0)
value: (big numpy array)
size: (4, 256, 256, 3)

It seems that split_1 and ParseSingleSequenceExample are not actually receiving any data, and thus cause this squeeze error since there is nothing to squeeze. But resize image still gets data.
Additionally, if I ONLY fetch ResizeImage/resize_images/ResizeBilinear:0, I can fetch it a couple of times (repeatedly fetching in a loop), and then it fails. Perhaps the model fails after one batch?

I'm not sure if this counts a duplicate, but here are some related threads:
https://github.com/tensorflow/models/issues/6027
https://github.com/tensorflow/models/issues/5869
https://stackoverflow.com/questions/54093931/lstm-object-detection-tensorflow

I've also emailed the authors and heard nothing back.

EDIT:

I should mention, I removed ssd_random_crop from data augmentation options in the config because it was giving me an error "the function ssd_random_crop requires argument groundtruth_weights"
Not sure if this would matter at all

Source code / logs

I tried two ways of creating tfrecords. The first was taken from tf_sequence_example_decoder_test.py, in this repo. The only change was swapping to sequences of length 4 to match the config file.

writer = tf.python_io.TFRecordWriter(path)
with tf.Session() as sess:
    for _ in range(2000):
        image_tensor = np.random.randint(255, size=(16, 16, 3)).astype(np.uint8)
        print(image_tensor)

        encoded_jpeg = tf.image.encode_jpeg(tf.constant(image_tensor)).eval()

        sequence_example = example_pb2.SequenceExample(
            context=feature_pb2.Features(
                feature={
                    'image/format':
                        feature_pb2.Feature(
                            bytes_list=feature_pb2.BytesList(
                                value=['jpeg'.encode('utf-8')])),
                    'image/height':
                        feature_pb2.Feature(
                            int64_list=feature_pb2.Int64List(value=[16])),
                    'image/width':
                        feature_pb2.Feature(
                            int64_list=feature_pb2.Int64List(value=[16])),
                }),
            feature_lists=feature_pb2.FeatureLists(
                feature_list={
                    'image/encoded':
                        feature_pb2.FeatureList(feature=[
                            feature_pb2.Feature(
                                bytes_list=feature_pb2.BytesList(
                                    value=[encoded_jpeg])), feature_pb2.Feature(
                                bytes_list=feature_pb2.BytesList(
                                    value=[encoded_jpeg])), feature_pb2.Feature(
                                bytes_list=feature_pb2.BytesList(
                                    value=[encoded_jpeg])), feature_pb2.Feature(
                                bytes_list=feature_pb2.BytesList(
                                    value=[encoded_jpeg]))
                        ]),
                    'image/object/bbox/xmin':
                        feature_pb2.FeatureList(feature=[
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0]))
                        ]),
                    'image/object/bbox/xmax':
                        feature_pb2.FeatureList(feature=[
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0]))
                        ]),
                    'image/object/bbox/ymin':
                        feature_pb2.FeatureList(feature=[
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[0.0]))
                        ]),
                    'image/object/bbox/ymax':
                        feature_pb2.FeatureList(feature=[
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0])),
                            feature_pb2.Feature(
                                float_list=feature_pb2.FloatList(value=[1.0]))
                        ]),
                    'image/object/class/label':
                        feature_pb2.FeatureList(feature=[
                            feature_pb2.Feature(
                                int64_list=feature_pb2.Int64List(value=[1])),
                            feature_pb2.Feature(
                                int64_list=feature_pb2.Int64List(value=[1])),
                            feature_pb2.Feature(
                                int64_list=feature_pb2.Int64List(value=[1])),
                            feature_pb2.Feature(
                                int64_list=feature_pb2.Int64List(value=[1]))
                        ]),
                }))

        writer.write(sequence_example.SerializeToString())
writer.close()

I also tried adapting a method I found here: https://github.com/wakanda-ai/tf-detectors
For this I used a couple sample xml files in PASCAL VOC format from a training set I have for one of the normal object_detection models.

    # Iterate frames
    for data, img_path in zip(dicts, imgs_path):
        ## open single frame
        with tf.gfile.FastGFile(img_path, 'rb') as fid:
            encoded_jpg = fid.read()
        encoded_jpg_io = io.BytesIO(encoded_jpg)
        image = Image.open(encoded_jpg_io)
        if image.format != 'JPEG':
            raise ValueError('Image format not JPEG')
        key = hashlib.sha256(encoded_jpg).hexdigest()

        ## validation
        assert int(data['size']['height']) == height
        assert int(data['size']['width']) == width

        ## iterate objects
        xmin, ymin = [], []
        xmax, ymax = [], []
        name = []
        classval =  []
        occluded = []
        generated = []
        if 'object' in data:
            for obj in data['object']:
                xmin.append(float(obj['bndbox']['xmin']) / width)
                ymin.append(float(obj['bndbox']['ymin']) / height)
                xmax.append(float(obj['bndbox']['xmax']) / width)
                ymax.append(float(obj['bndbox']['ymax']) / height)
                name.append(obj['name'].encode('utf8'))
                classval.append(1)
                occluded.append(0)
                generated.append(0)
        else:
            xmin.append(float(-1))
            ymin.append(float(-1))
            xmax.append(float(-1))
            ymax.append(float(-1))
            name.append('NoObject'.encode('utf8'))
            classval.append(1)
            occluded.append(0)
            generated.append(0)

        ## append tf_feature to list
        filenames.append(dataset_util.bytes_feature(data['filename'].encode('utf8')))
        encodeds.append(dataset_util.bytes_feature(encoded_jpg))
        sources.append(dataset_util.bytes_feature(data['source']['database'].encode('utf8')))
        keys.append(dataset_util.bytes_feature(key.encode('utf8')))
        formats.append(dataset_util.bytes_feature('jpeg'.encode('utf8')))
        xmins.append(dataset_util.float_list_feature(xmin))
        ymins.append(dataset_util.float_list_feature(ymin))
        xmaxs.append(dataset_util.float_list_feature(xmax))
        ymaxs.append(dataset_util.float_list_feature(ymax))
        names.append(dataset_util.bytes_list_feature(name))
        occludeds.append(dataset_util.int64_list_feature(occluded))
        generateds.append(dataset_util.int64_list_feature(generated))
        class_labels.append(dataset_util.int64_list_feature(classval))

    # Non sequential features
    context = tf.train.Features(feature={
        'video/folder': dataset_util.bytes_feature(folder.encode('utf8')),
        'video/frame_number': dataset_util.int64_feature(len(imgs_path)),
        'video/height': dataset_util.int64_feature(height),
        'video/width': dataset_util.int64_feature(width),
        })
    # Sequential features
    tf_feature_lists = {
        'image/filename': tf.train.FeatureList(feature=filenames),
        'image/encoded': tf.train.FeatureList(feature=encodeds),
        'image/sources': tf.train.FeatureList(feature=sources),
        'image/key/sha256': tf.train.FeatureList(feature=keys),
        'image/format': tf.train.FeatureList(feature=formats),
        'image/object/bbox/xmin': tf.train.FeatureList(feature=xmins),
        'image/object/bbox/xmax': tf.train.FeatureList(feature=xmaxs),
        'image/object/bbox/ymin': tf.train.FeatureList(feature=ymins),
        'image/object/bbox/ymax': tf.train.FeatureList(feature=ymaxs),
        'image/object/class/text': tf.train.FeatureList(feature=names),
        'image/object/class/label': tf.train.FeatureList(feature=class_labels),
        'image/object/occluded': tf.train.FeatureList(feature=occludeds),
        'image/object/generated': tf.train.FeatureList(feature=generateds),
        }
    feature_lists = tf.train.FeatureLists(feature_list=tf_feature_lists)
    # Make single sequence example
    tf_example = tf.train.SequenceExample(context=context, feature_lists=feature_lists)
    return tf_example

Tfrecords created with both of these approaches yielded identical errors.