Can't run preprocessing

After reproducing paper results, I am interested in the model's (ITM) performance on different datasets. To achieve that I have to run the preprocessing pipeline on my own images but I am getting errors on all different GPU setups I have attempted it on (K-80 | V100 | A100) because of CUDA version issues. Could you please help me troubleshoot? Perhaps switching versions of nvidia/cuda on the host would help?

You can see the stacktrace below, this particular host machine has 8 V100 GPUS, CUDA 10.2 and nvidia-driver 440.

```
$ bash scripts/extract_imgfeat.sh $PATH_TO_IMG $PATH_TO_NP
extracting image features...

Status: Downloaded newer image for chenrocks/butd-caffe:nlvr2

==================
== NVIDIA Caffe ==
==================

NVIDIA Release 19.08 (build 7603994)
NVIDIA Caffe Version 0.17.3

Container image Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
Copyright (c) 2014, 2015, The Regents of the University of California (Regents)
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.
Use 'nvidia-docker run' to start this container; see
https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker .

NOTE: MOFED driver for multi-node communication was not detected.
Multi-node communication performance may be reduced.

/src/tools/../lib/fast_rcnn/config.py:288: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
yaml_cfg = edict(yaml.load(f))
I0127 11:23:48.148620 1 _caffe.cpp:64] Using devices [0]
Called with args:
Namespace(caffemodel=None, caffemodelDir='./data/faster_rcnn_models/', cfg_file='./experiments/cfgs/faster_rcnn_end2end_resnet.yml', file_id=-1, gpu_id='0', outfile=None, prefix='nlvr2', prototxt='./models/vg/ResNet-101/faster_rcnn_end2end_final/test.prototxt')
Using config:
{'DATA_DIR': '/src/data',
'DEDUP_BOXES': 0.0625,
'EPS': 1e-14,
'EXP_DIR': 'faster_rcnn_resnet',
'GPU_ID': 0,
'MATLAB': 'matlab',
'MODELS_DIR': '/src/models/pascal_voc',
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'RNG_SEED': 3,
'ROOT_DIR': '/src',
'TEST': {'AGNOSTIC': False,
'BBOX_REG': True,
'HAS_ATTRIBUTES': True,
'HAS_RELATIONS': False,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'NMS': 0.3,
'PROPOSAL_METHOD': 'selective_search',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'SCALES': [600],
'SOFT_NMS': 0,
'SVM': False},
'TRAIN': {'AGNOSTIC': False,
'ASPECT_GROUPING': True,
'BATCH_SIZE': 64,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'FG_FRACTION': 0.5,
'FG_THRESH': 0.5,
'HAS_ATTRIBUTES': True,
'HAS_RELATIONS': False,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'MAX_SIZE': 1000,
'MIN_RELATION_FRACTION': 0.25,
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 64,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 16,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'RPN_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'RPN_NORMALIZE_TARGETS': False,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_INFIX': '',
'SNAPSHOT_ITERS': 10000,
'USE_FLIPPED': True,
'USE_PREFETCH': False},
'USE_GPU_NMS': False}
./data/faster_rcnn_models/resnet101_faster_rcnn_final.caffemodel
E0127 11:23:48.156255 107 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.
E0127 11:23:48.156461 107 common.cpp:121] Cannot create Curand generator. Curand won't be available.
F0127 11:23:48.156692 107 common.cpp:152] Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version
*** Check failure stack trace: ***
@ 0x7fe4481b00cd google::LogMessage::Fail()
@ 0x7fe4481b1f33 google::LogMessage::SendToLog()
@ 0x7fe4481afc28 google::LogMessage::Flush()
@ 0x7fe4481b2999 google::LogMessageFatal::~LogMessageFatal()
@ 0x7fe4485ac742 caffe::Caffe::SetDevice()
@ 0x7fe4493836b1 boost::python::objects::caller_py_function_impl<>::operator()()
@ 0x7fe4478f7e55 boost::python::objects::function::call()
@ 0x7fe4478f8008 (unknown)
@ 0x7fe4478fe9a3 boost::python::handle_exception_impl()
@ 0x7fe4478f56c9 (unknown)
@ 0x55c10b7ffe30 (unknown)
@ 0x55c10b7f852a (unknown)
@ 0x55c10b81403c (unknown)
@ 0x55c10b7e3f1e (unknown)
@ 0x55c10b7fd2d5 (unknown)
@ 0x55c10b7ffbe2 (unknown)
@ 0x55c10b7ffbe2 (unknown)
@ 0x55c10b7f852a (unknown)
@ 0x55c10b813d99 (unknown)
@ 0x55c10b82c95e (unknown)
@ 0x55c10b82c56a (unknown)
@ 0x55c10b7e905b (unknown)
@ 0x55c10b7ffe30 (unknown)
@ 0x55c10b7ffbe2 (unknown)
@ 0x55c10b7f852a (unknown)
@ 0x55c10b7f7fb9 (unknown)
@ 0x55c10b828e7f (unknown)
@ 0x55c10b823c12 (unknown)
@ 0x55c10b82309d (unknown)
@ 0x55c10b7d1d6b (unknown)
@ 0x7fe4f0f52b97 __libc_start_main
@ 0x55c10b7d15ea (unknown)
done

```

Start with scripts/extract_imgfeat.sh and the chenrocks/butd-caffe:nlvr2 container referenced in the stack trace. Check how the preprocessing command exposes the GPU, then review the reported driver/runtime mismatch and the configuration path in lib/fast_rcnn/config.py. Done means the image-feature preprocessing command runs successfully on a supported GPU setup.

Written by the indexing model from the issue text.