tensorflow / tensorflow/models
offline_eval_map_corloc not updated for changes in config_util
@pkulzc is already working on this.
Since Jun 24, 2020.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
System information
- What is the top-level directory of the model you are using: models\research\object_detection
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
- Output of tf_env_collect.sh (abridged):
== check python ===================================================
python version: 3.5.2
python branch:
python build version: ('default', 'Nov 12 2018 13:43:14')
python compiler version: GCC 5.4.0 20160609
python implementation: CPython
== check os platform ===============================================
os: Linux
os kernel version: #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019
os release version: 4.15.0-55-generic
os platform: Linux-4.15.0-55-generic-x86_64-with-Ubuntu-16.04-xenial
linux distribution: ('Ubuntu', '16.04', 'xenial')
linux os distribution: ('Ubuntu', '16.04', 'xenial')
mac version: ('', ('', '', ''), '')
uname: uname_result(system='Linux', node='7ff2d67273eb', release='4.15.0-55-generic', version='#60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019', machine='x86_64', processor='x86_64')
architecture: ('64bit', '')
machine: x86_64
== are we in docker =============================================
Yes
== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
== check pips ===================================================
numpy 1.14.5
protobuf 3.7.1
tensorflow-estimator 1.13.0
tensorflow-gpu 1.13.1+nv
== check for virtualenv =========================================
False
== tensorflow import ============================================
tf.version.VERSION = 1.13.1
tf.version.GIT_VERSION = b'unknown'
tf.version.COMPILER_VERSION = 5.4.0 20160609
== env ==========================================================
LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/lib/tensorflow
DYLD_LIBRARY_PATH is unset
== nvidia-smi ===================================================
Thu Aug 8 20:56:03 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.40 Driver Version: 430.40 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... Off | 00000000:3B:00.0 Off | 0 |
| N/A 38C P0 36W / 250W | 0MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
== cuda libs ===================================================
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart.so.10.1.163
== tensorflow installed from info ==================
== bazel version ===============================================
Build label: 0.19.2
Build time: Mon Nov 19 16:25:09 2018 (1542644709)
Build timestamp: 1542644709
Build timestamp as int: 1542644709
- Exact command to reproduce:
python /tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py \
--eval_dir=/path/to/output \
--eval_config_path=/path/to/eval_config.pbtxt \
--input_config_path=/path/to/input_config.pbtxt
Describe the problem
I am attempting to evaluate the object detection results of a retrained model (SSD Inception v2 COCO) following the Inference and evaluation on the Open Images dataset tutorial. I believe there is a bug in the offline_eval_map_corloc.py script. When I use it to evaluate my results with the following command (paths replaced with placeholders)
python /tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py \
--eval_dir=/path/to/output \
--eval_config_path=/path/to/eval_config.pbtxt \
--input_config_path=/path/to/input_config.pbtxt
I get the result
WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
* https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
* https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.
Traceback (most recent call last):
File "/tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py", line 171, in <module>
tf.app.run(main)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/tensorflow/models/research/object_detection/metrics/offline_eval_map_corloc.py", line 162, in main
input_config = configs['eval_input_config']
KeyError: 'eval_input_config'
After examining config_util.py I noticed that get_configs_from_multiple_files now returns the input config in a list and the key has been changed from eval_input_config to eval_input_configs.
This is the exact same issue as here which is closed, but the bug was never actually addressed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.