facebookresearch / facebookresearch/detectron2

detection_utils.read_image inconsistent with cv2.imread

Open
#2,379 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

## Instructions To Reproduce the Issue:

In the DefaultPredictor documentation, it is recommended to run inference using
```
pred = DefaultPredictor(cfg)
inputs = cv2.imread("input.jpg")
outputs = pred(inputs)
```
However, using cv2.imread produces a different ndarray than if the image is loaded via the default DatasetMapper, which uses detectron2.data.detection_utils.read_image.
```
image = utils.read_image(dataset_dict["file_name"], format=self.image_format)
```
So this causes an inconsistency if models are trained with DefaultTrainer and use DefaultPredictor for inference.
Evaluation during training with DefaultTrainer, which uses EvalHook -> DefaultTrainer.test -> build_detection_test_loader -> DatasetMapper, will load different image arrays, and thus result in different predictions than the recommended DefaultPredictor commands. This seems like unintended behavior based on the documentation.

1. Full runnable code or full changes you made:
```
import cv2
from detectron2.data.detection_utils import read_image

image_path = "path/to/attached/file.jpg"

# these should be the same
cv2_im = cv2.imread(image_path)
rd_im = read_image(image_path, format="BGR")
print(cv2_im == rd_im)
```
```
[[[ True True True]
[ True True True]
[ True True True]
...
[ True True True]
[ True True True]
[ True True True]]

[[ True True True]
[ True True True]
[ True True True]
...
[ True True True]
[ True True True]
[ True True True]]

[[ True True True]
[ True True True]
[ True True True]
...
[ True True True]
[ True True True]
[ True True True]]

...

[[False False False]
[False True False]
[False False False]
...
[False False False]
[False False False]
[False True False]]

[[False False False]
[False True False]
[False True False]
...
[False True False]
[False True False]
[False False False]]

[[False False False]
[False False False]
[False False False]
...
[False True False]
[False True False]
[False True False]]]
```

## Environment:

Provide your environment information using the following command:
```
---------------------- ---------------------------------------------------------------------------------------
sys.platform linux
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) [GCC 7.5.0]
numpy 1.19.2
detectron2 0.3 @/home/ubuntu/anaconda3/envs/aifish/lib/python3.7/site-packages/detectron2
Compiler GCC 7.3
CUDA compiler CUDA 10.1
detectron2 arch flags 3.7, 5.0, 5.2, 6.0, 6.1, 7.0, 7.5
DETECTRON2_ENV_MODULE
PyTorch 1.6.0+cu101 @/home/ubuntu/anaconda3/envs/aifish/lib/python3.7/site-packages/torch
PyTorch debug build False
GPU available True
GPU 0 Tesla V100-SXM2-16GB (arch=7.0)
CUDA_HOME /usr/local/cuda
Pillow 7.2.0
torchvision 0.7.0+cu101 @/home/ubuntu/anaconda3/envs/aifish/lib/python3.7/site-packages/torchvision
torchvision arch flags 3.5, 5.0, 6.0, 7.0, 7.5
fvcore 0.1.2.post20200910
cv2 4.4.0
---------------------- ---------------------------------------------------------------------------------------
PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2019.0.5 Product Build 20190808 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 10.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
- CuDNN 7.6.3
- Magma 2.5.2
- Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,
```
Attached image:
![5858bf1e-23d2-11e8-a6a3-ec086b02610b](https://user-images.githubusercontent.com/1168887/101944291-19eb4a80-3bb2-11eb-92a1-56d3ee3db49a.jpg)

Contributor guide

Open the contributing guide

Research direction

Start with detection_utils.read_image and trace the DatasetMapper and DefaultPredictor paths described in the issue. Run the supplied cv2.imread versus read_image comparison, then verify that the documented training and inference paths produce consistent image arrays and predictions.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, opencv, python
Domain
computer-vision
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.