Object Detection API: NaN loss if label map's item name doesn't match tf record's image/object/class/text value

System information

• What is the top-level directory of the model you are using: object_detection

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04

• TensorFlow installed from (source or binary): NVIDIA Docker container nvcr.io/nvidia/tensorflow:19.06-py3

• TensorFlow version (use command below): 1.13.1

• Bazel version (if compiling from source): n/a

• CUDA/cuDNN version: 10.1/unknown

• GPU model and memory: 2 RTX 2080 Ti, 11GBs each

• Exact command to reproduce: see below

• Models version: cloned from master, commit d09994b26ac8592765be45dc8a8444ead55eb32b

Describe the problem

Given a single-class custom-generated TF Record dataset (the code I use for the generation is below) to use for fine-tuning a Faster R-CNN Inception V2 object detection net, the training works fine if the label map item's name property matches exactly the value I use for image/object/class/text in the tf record and crashes (repeatably) if I change the name to anything else (pretrained weights from the model zoo, pipeline.config trom the tar.gz altered in batch size and number of classes).

Training is done via the legacy/train.py script (since model_main doesn't support distributed training).

Note: I can't currently provide a reproducible example, as I can't share my dataset. Given the observed behaviour, I believe the issue is not specfic to the dataset itself (I tested with multiple versions of the dataset, some known-good from previous trainings that break when I alter the label map) and could be reproduced on any standard dataset, but right now I don't have a chance to verify this.

Note: I encountered this bug while experimenting with the new remap_labels data augmentation option, so I'll cc @pkulzc as it would seem he's the author of the merge commit that introduced it, while the feature itself is from Zhichao Lu, as this might be of interest (do remapped records work, if they are remapped to an existing label with a different name?)

Source code / logs

Relevant functions for the dataset generation:

def create_tf_example(filename, encoded_image_data, polys_dict):
  height = 672
  width = 672
  image_format = b'png'

  bboxes = [...] # get bboxess from polygons

  xmins = bboxes[...,0]/width
  ymins = bboxes[...,1]/height
  xmaxs = bboxes[...,2]/width
  ymaxs = bboxes[...,3]/height

  classes_text = [b'my_class']*len(bboxes)
  classes = [1]*len(bboxes)

  feature_dict = {
      'image/height': dataset_util.int64_feature(height),
      'image/width': dataset_util.int64_feature(width),
      'image/filename': dataset_util.bytes_feature(filename.encode('utf-8')),
      'image/source_id': dataset_util.bytes_feature(filename.encode('utf-8')),
      'image/encoded': dataset_util.bytes_feature(encoded_image_data),
      'image/format': dataset_util.bytes_feature(image_format),
      'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
      'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
      'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
      'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
      'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
      'image/object/class/label': dataset_util.int64_list_feature(classes)
  }

  tf_example = tf.train.Example(features=tf.train.Features(feature=feature_dict))
  return tf_example

def make_dataset(files, output_filebase, num_shards=5):
  import contextlib2
  from object_detection.dataset_tools import tf_record_creation_util
  from random import shuffle

  os.makedirs(os.path.dirname(output_filebase), exist_ok=True)

  examples = []
  for fn in files:
    with open(fn, 'rb') as fp:
      im_data = fp.read()

    [...] # read polygons for image and store them in polys_dict

    examples.append(create_tf_example(fn, im_data, polys_dict))
  shuffle(examples)
  
  with contextlib2.ExitStack() as tf_record_close_stack:
    output_tfrecords = tf_record_creation_util.open_sharded_output_tfrecords(
        tf_record_close_stack, output_filebase, num_shards)
    for index, tf_example in enumerate(examples):
      output_shard_index = index % num_shards
      output_tfrecords[output_shard_index].write(tf_example.SerializeToString())

The label map contains the following:

item {
    id: 1
    name: 'my_class'
}