KeyError: 'groundtruth_verified_neg_classes'

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py

2. Describe the bug

This error occurs when training is started using the code from model_main_tf2.py

  File "main.py", line 197, in main
    tf.compat.v1.app.run(main=training)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "main.py", line 103, in training
    record_summaries=args.record_summaries)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/object_detection/model_lib_v2.py", line 526, in train_loop
    train_dataset_fn)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py", line 1128, in experimental_distribute_datasets_from_function
    dataset_fn, options)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/distribute/mirrored_strategy.py", line 503, in _experimental_distribute_datasets_from_function
    self._container_strategy())
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/distribute/input_lib.py", line 136, in get_distributed_datasets_from_function
    strategy)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/distribute/input_lib.py", line 1183, in __init__
    dataset_fn))
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/distribute/input_lib.py", line 1767, in _create_datasets_per_worker_with_input_context
    dataset = dataset_fn(ctx)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/object_detection/model_lib_v2.py", line 521, in train_dataset_fn
    input_context=input_context)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/object_detection/inputs.py", line 851, in train_input
    `input_fn` for `Estimator` in EVAL mode.
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/object_detection/builders/dataset_builder.py", line 243, in build
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/object_detection/builders/dataset_builder.py", line 222, in dataset_map_fn
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2564, in map_with_legacy_function
    use_legacy_function=True))
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 4084, in __init__
    use_legacy_function=use_legacy_function)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3339, in __init__
    self._function.add_to_graph(ops.get_default_graph())
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/framework/function.py", line 544, in add_to_graph
    self._create_definition_if_needed()
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/framework/function.py", line 376, in _create_definition_if_needed
    self._create_definition_if_needed_impl()
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/framework/function.py", line 407, in _create_definition_if_needed_impl
    capture_resource_var_by_value=self._capture_resource_var_by_value)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/framework/function.py", line 969, in func_graph_from_py_func
    outputs = func(*func_graph.inputs)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3331, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 3299, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "/databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 258, in wrapper
    raise e.ag_error_metadata.to_exception(e)
tensorflow.python.autograph.pyct.error_utils.KeyError: in user code:

    /databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/object_detection/inputs.py:818 transform_and_pad_input_data_fn  *
        tensor_dict = pad_input_data_to_static_shapes(
    /databricks/conda/envs/databricks-ml-gpu/lib/python3.7/site-packages/object_detection/inputs.py:523 pad_input_data_to_static_shapes  *
        padded_tensor_dict[tensor_name] = shape_utils.pad_or_clip_nd(

    KeyError: 'groundtruth_verified_neg_classes'

3. Steps to reproduce

Clone the latest version of models and start the training using the mobilenet ssd v2 pipeline config.

4. Expected behavior

Start the training procedure

5. Additional context

I manage to fix this problem using the 24e41ffe97c601e52b35682170e7abceed0eae1a commit on master branch, so this is probably due to a recent commit on master branch.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• TensorFlow installed from (source or binary): 2.3.1
• TensorFlow version (use command below): 2.3.1
• Python version: 3.7