Eval error: UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2(using tf1 and a model from the garden for tf1).
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/model_main.py

2. Describe the bug

Evaluation doesn't accept unicode characters but training does

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/PIL/ImageFont.py", line 128, in getsize
    return self.font.getsize(text)

UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)


     [[node map_1/while/PyFunc (defined at /home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
     [[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_16/NonMaxSuppressionV5/_2543]]
0 successful operations.
0 derived errors ignored.

3. Steps to reproduce

While training with a custom dataset that contains unicode characters the training works ok till a checkpoint is saved and evaluated then this error shows.
run training with custom dataset using model_train.py that has Arabic labels (tf1.15)

4. Expected behavior

Expected training to run through all steps and not crashing at checkpoint evaluation.

5. Additional context

2020-09-23 17:42:02.046396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-09-23 17:42:02.051500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-23 17:42:02.051549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2020-09-23 17:42:02.051758: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2020-09-23 17:42:02.054590: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 17:42:02.054801: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 17:42:02.054985: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6738 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from /home/mai/ssd_ckpt/tf1_fine_tuning_vehicle/6.25k_steps_frz_var/train/model.ckpt-872
I0923 17:42:02.059047 139873538410304 saver.py:1284] Restoring parameters from /home/mai/ssd_ckpt/tf1_fine_tuning_vehicle/6.25k_steps_frz_var/train/model.ckpt-872
INFO:tensorflow:Running local_init_op.
I0923 17:42:03.378439 139873538410304 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0923 17:42:03.507816 139873538410304 session_manager.py:502] Done running local_init_op.
2020-09-23 17:42:05.991016: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)
Traceback (most recent call last):

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
    ret = func(*args)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 407, in visualization_py_func_fn
    **kwargs)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 1239, in visualize_boxes_and_labels_on_image_array
    use_normalized_coordinates=use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 162, in draw_bounding_box_on_image_array
    use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in draw_bounding_box_on_image
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in <listcomp>
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/PIL/ImageFont.py", line 128, in getsize
    return self.font.getsize(text)

UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)


Traceback (most recent call last):
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)
Traceback (most recent call last):

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
    ret = func(*args)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 407, in visualization_py_func_fn
    **kwargs)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 1239, in visualize_boxes_and_labels_on_image_array
    use_normalized_coordinates=use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 162, in draw_bounding_box_on_image_array
    use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in draw_bounding_box_on_image
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in <listcomp>
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/PIL/ImageFont.py", line 128, in getsize
    return self.font.getsize(text)

UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)


	 [[{{node map_1/while/PyFunc}}]]
  (1) Invalid argument: UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)
Traceback (most recent call last):

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
    ret = func(*args)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 407, in visualization_py_func_fn
    **kwargs)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 1239, in visualize_boxes_and_labels_on_image_array
    use_normalized_coordinates=use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 162, in draw_bounding_box_on_image_array
    use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in draw_bounding_box_on_image
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in <listcomp>
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/PIL/ImageFont.py", line 128, in getsize
    return self.font.getsize(text)

UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)


	 [[{{node map_1/while/PyFunc}}]]
	 [[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_20/NonMaxSuppressionV5/_2561]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./object_detection/model_main.py", line 119, in <module>
    tf.app.run()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "./object_detection/model_main.py", line 104, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1426, in run
    run_metadata=run_metadata))
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run
    if self._save(run_context.session, global_step):
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save
    if l.after_save(session, step):
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 519, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate
    name=name)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval
    return _evaluate()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 511, in _evaluate
    output_dir=self.eval_dir(name))
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1619, in _evaluate_run
    config=self._session_config)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/evaluation.py", line 272, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
    run_metadata=run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)
Traceback (most recent call last):

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
    ret = func(*args)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 407, in visualization_py_func_fn
    **kwargs)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 1239, in visualize_boxes_and_labels_on_image_array
    use_normalized_coordinates=use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 162, in draw_bounding_box_on_image_array
    use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in draw_bounding_box_on_image
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in <listcomp>
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/PIL/ImageFont.py", line 128, in getsize
    return self.font.getsize(text)

UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)


	 [[node map_1/while/PyFunc (defined at /home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
  (1) Invalid argument: UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)
Traceback (most recent call last):

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
    ret = func(*args)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 407, in visualization_py_func_fn
    **kwargs)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 1239, in visualize_boxes_and_labels_on_image_array
    use_normalized_coordinates=use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 162, in draw_bounding_box_on_image_array
    use_normalized_coordinates)

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in draw_bounding_box_on_image
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 219, in <listcomp>
    display_str_heights = [font.getsize(ds)[1] for ds in display_str_list]

  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/PIL/ImageFont.py", line 128, in getsize
    return self.font.getsize(text)

UnicodeEncodeError: 'latin-1' codec can't encode character '\u062d' in position 0: ordinal not in range(256)


	 [[node map_1/while/PyFunc (defined at /home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
	 [[Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_20/NonMaxSuppressionV5/_2561]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'map_1/while/PyFunc':
  File "./object_detection/model_main.py", line 119, in <module>
    tf.app.run()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "./object_detection/model_main.py", line 104, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1494, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1259, in run
    run_metadata=run_metadata)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/monitored_session.py", line 1426, in run
    run_metadata=run_metadata))
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/basic_session_run_hooks.py", line 594, in after_run
    if self._save(run_context.session, global_step):
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/training/basic_session_run_hooks.py", line 619, in _save
    if l.after_save(session, step):
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 519, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 539, in _evaluate
    self._evaluator.evaluate_and_export())
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 920, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 480, in evaluate
    name=name)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 522, in _actual_eval
    return _evaluate()
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 504, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1511, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1547, in _call_model_fn_eval
    features, labels, ModeKeys.EVAL, config)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/mai/models/research/object_detection/model_lib.py", line 620, in model_fn
    eval_dict)
  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 1426, in get_estimator_eval_metric_ops
    images = self.images_from_evaluation_dict(eval_dict)
  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 1498, in images_from_evaluation_dict
    self._keypoint_edges)
  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 752, in draw_side_by_side_evaluation_image
    use_normalized_coordinates=use_normalized_coordinates)
  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 621, in draw_bounding_boxes_on_image_tensors
    images = tf.map_fn(draw_boxes, elems, dtype=tf.uint8, back_prop=False)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/map_fn.py", line 268, in map_fn
    maximum_iterations=n)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2753, in while_loop
    return_same_structure)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2245, in BuildLoop
    pred, body, original_loop_vars, loop_vars, shape_invariants)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2170, in _BuildLoop
    body_result = body(*packed_vars_for_body)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 2705, in <lambda>
    body = lambda i, lv: (i + 1, orig_body(*lv))
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/map_fn.py", line 257, in compute
    packed_fn_values = fn(packed_values)
  File "/home/mai/models/research/object_detection/utils/visualization_utils.py", line 618, in draw_boxes
    tf.uint8)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 513, in py_func
    return py_func_common(func, inp, Tout, stateful, name=name)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 495, in py_func_common
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 318, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_script_ops.py", line 170, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/home/mai/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 186.04):
• Mobile device name if the issue happens on a mobile device:
• TensorFlow installed from (source or binary): binary using conda install
• TensorFlow version (use command below): tensorflow-gpu==1.15
• Python version: 3.6
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 10.0/7.6.5
• GPU model and memory: GeForce RTX 2060 super 8 GB