NotImplementedError: Cannot convert a symbolic Tensor (strided_slice:0) to a numpy array.

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py

2. Describe the bug

I am trying to train object detection model for custom data using tutorial on link. I tested the for environment faults using https://github.com/tensorflow/models/blob/master/research/object_detection/builders/model_builder_tf2_test.py. All test passed. But when I put it for training the models gives out error.
Logs:

2021-02-05 14:26:00.620416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-02-05 14:26:03.557625: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-02-05 14:26:03.790381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0002:00:00.0 name: Tesla M60 computeCapability: 5.2
coreClock: 1.1775GHz coreCount: 16 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 149.31GiB/s
2021-02-05 14:26:03.796912: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-02-05 14:26:03.805410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-02-05 14:26:03.813387: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-02-05 14:26:03.817933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-02-05 14:26:03.827291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-02-05 14:26:03.834728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-02-05 14:26:03.850086: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-02-05 14:26:03.857292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-02-05 14:26:03.860948: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-02-05 14:26:03.875535: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x80ec5ff940 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-02-05 14:26:03.880482: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-02-05 14:26:03.885168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0002:00:00.0 name: Tesla M60 computeCapability: 5.2
coreClock: 1.1775GHz coreCount: 16 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 149.31GiB/s
2021-02-05 14:26:03.891972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-02-05 14:26:03.895720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-02-05 14:26:03.899501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-02-05 14:26:03.904606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-02-05 14:26:03.908422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-02-05 14:26:03.912343: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-02-05 14:26:03.916413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-02-05 14:26:03.924000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-02-05 14:26:04.700603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength
1 edge matrix:
2021-02-05 14:26:04.704509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0
2021-02-05 14:26:04.706737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N
2021-02-05 14:26:04.726496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7048 MB memory) -> physical GPU (device: 0, name: Tesla M60, pci bus id: 0002:00:00.0, compute capability: 5.2)
2021-02-05 14:26:04.736932: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x810d9b6950 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-02-05 14:26:04.741982: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla M60, Compute Capability 5.2
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
I0205 14:26:04.749317  8188 mirrored_strategy.py:500] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: None
I0205 14:26:04.755326  8188 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0205 14:26:04.755326  8188 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Reading unweighted datasets: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
I0205 14:26:04.923312  8188 dataset_builder.py:163] Reading unweighted datasets: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
INFO:tensorflow:Reading record datasets for input file: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
I0205 14:26:04.926311  8188 dataset_builder.py:80] Reading record datasets for input file: ['E:/DS_2020_Wildlife/Multi_Class_Classification/Tensorflow/workspace/annotations/train.record']
INFO:tensorflow:Number of filenames to read: 1
I0205 14:26:04.926311  8188 dataset_builder.py:81] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0205 14:26:04.926311  8188 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
W0205 14:26:04.928313  8188 deprecation.py:317] From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:101: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_deterministic`.
WARNING:tensorflow:From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0205 14:26:04.952313  8188 deprecation.py:317] From E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is
deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
Traceback (most recent call last):
  File "model_main_tf2.py", line 115, in <module>
    tf.compat.v1.app.run()
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\platform\app.py", line 40,
in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\absl\app.py", line 300, in run
    _run_main(main, args)
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 106, in main
    model_lib_v2.train_loop(
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\model_lib_v2.py", line 569,
in train_loop
    load_fine_tune_checkpoint(detection_model,
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\model_lib_v2.py", line 352,
in load_fine_tune_checkpoint
    features, labels = iter(input_dataset).next()
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 858, in __iter__
    iterators, element_spec = _create_iterators_per_worker_with_input_context(
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\distribute\input_lib.py", line 1401, in _create_iterators_per_worker_with_input_context
    dataset = dataset_fn(ctx)
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\model_lib_v2.py", line 521,
in train_dataset_fn
    train_input = inputs.train_input(
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\inputs.py", line 893, in train_input
    dataset = INPUT_BUILDER_UTIL_MAP['dataset_build'](
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py", line 251, in build
    dataset = dataset_map_fn(dataset, decoder.decode, batch_size,
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\builders\dataset_builder.py", line 236, in dataset_map_fn
    dataset = dataset.map_with_legacy_function(
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\util\deprecation.py", line
324, in new_func
    return func(*args, **kwargs)
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 2402, in map_with_legacy_function
    ParallelMapDataset(
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 4016, in __init__
    self._map_func = StructuredFunctionWrapper(
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3196, in __init__
    self._function.add_to_graph(ops.get_default_graph())
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 544, in add_to_graph
    self._create_definition_if_needed()
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 376, in _create_definition_if_needed
    self._create_definition_if_needed_impl()
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 398, in _create_definition_if_needed_impl
    temp_graph = func_graph_from_py_func(
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\function.py", line 969, in func_graph_from_py_func
    outputs = func(*func_graph.inputs)
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3188, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3156, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 265, in wrapper
    raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:

    E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py:524 default_groundtruth_weights  *
        [tf.shape(tensor_dict[fields.InputDataFields.groundtruth_boxes])[0]],
    E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\ops\array_ops.py:2967 ones  **
        output = _constant_if_small(one, shape, dtype, name)
    E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\ops\array_ops.py:2662 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:5 prod

    E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\numpy\core\fromnumeric.py:3030 prod
        return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out,
    E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\numpy\core\fromnumeric.py:87 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    E:\DS_2020_Wildlife\Multi_Class_Classification\Tensorflow\venv\lib\site-packages\tensorflow\python\framework\ops.py:748 __array__
        raise NotImplementedError("Cannot convert a symbolic Tensor ({}) to a numpy"

    NotImplementedError: Cannot convert a symbolic Tensor (strided_slice:0) to a numpy array.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows Sever 2012
• TensorFlow installed from (source or binary): PIP
• TensorFlow version (use command below): 2.2
• Python version: 3.8
• CUDA/cuDNN version: 10.1
• GPU model and memory: Tesla M60