tensorflow / tensorflow/models

offline_eval_map_corloc not updated for changes in config_util

Open
#7,414 3 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Jun 24, 2020.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

System information
  • What is the top-level directory of the model you are using: models\research\object_detection
  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • Output of tf_env_collect.sh (abridged):

== check python ===================================================
python version: 3.5.2
python branch:
python build version: ('default', 'Nov 12 2018 13:43:14')
python compiler version: GCC 5.4.0 20160609
python implementation: CPython

== check os platform ===============================================
os: Linux
os kernel version: #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019
os release version: 4.15.0-55-generic
os platform: Linux-4.15.0-55-generic-x86_64-with-Ubuntu-16.04-xenial
linux distribution: ('Ubuntu', '16.04', 'xenial')
linux os distribution: ('Ubuntu', '16.04', 'xenial')
mac version: ('', ('', '', ''), '')
uname: uname_result(system='Linux', node='7ff2d67273eb', release='4.15.0-55-generic', version='#60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019', machine='x86_64', processor='x86_64')
architecture: ('64bit', '')
machine: x86_64

== are we in docker =============================================
Yes

== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

== check pips ===================================================
numpy 1.14.5
protobuf 3.7.1
tensorflow-estimator 1.13.0
tensorflow-gpu 1.13.1+nv

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.version.VERSION = 1.13.1
tf.version.GIT_VERSION = b'unknown'
tf.version.COMPILER_VERSION = 5.4.0 20160609

== env ==========================================================
LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/lib/tensorflow
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Thu Aug 8 20:56:03 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.40 Driver Version: 430.40 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... Off | 00000000:3B:00.0 Off | 0 |
| N/A 38C P0 36W / 250W | 0MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

== cuda libs ===================================================
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart.so.10.1.163

== tensorflow installed from info ==================

== bazel version ===============================================
Build label: 0.19.2
Build time: Mon Nov 19 16:25:09 2018 (1542644709)
Build timestamp: 1542644709
Build timestamp as int: 1542644709

  • Exact command to reproduce:
python /tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py \
	--eval_dir=/path/to/output \
	--eval_config_path=/path/to/eval_config.pbtxt \
	--input_config_path=/path/to/input_config.pbtxt
Describe the problem

I am attempting to evaluate the object detection results of a retrained model (SSD Inception v2 COCO) following the Inference and evaluation on the Open Images dataset tutorial. I believe there is a bug in the offline_eval_map_corloc.py script. When I use it to evaluate my results with the following command (paths replaced with placeholders)

python /tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py \
	--eval_dir=/path/to/output \
	--eval_config_path=/path/to/eval_config.pbtxt \
	--input_config_path=/path/to/input_config.pbtxt

I get the result

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

Traceback (most recent call last):
  File "/tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py", line 171, in <module>
    tf.app.run(main)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py", line 162, in main
    input_config = configs['eval_input_config']
KeyError: 'eval_input_config'

After examining config_util.py I noticed that get_configs_from_multiple_files now returns the input config in a list and the key has been changed from eval_input_config to eval_input_configs.

This is the exact same issue as here which is closed, but the bug was never actually addressed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.