tensorflow / tensorflow/models
Error when executing with gpu tutorial object detection.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
GPU-GeForce GTX 1650
driver nvidia- 450
CUDA-10.1
CUDNN-7.6
Tensoflow-2.2.0
python-3.8.3
Ubuntu-18.04
When I run the object detection tutorial, the gpu is correctly detects me and loads the model. The problem appears when running the inference.
in
1 for image_path in TEST_IMAGE_PATHS:
----> 2 show_inference(detection_model, image_path)
in show_inference(model, image_path)
4 image_np = np.array(Image.open(image_path))
5 # Actual detection.
----> 6 output_dict = run_inference_for_single_image(model, image_np)
7 # Visualization of the results of a detection.
8 vis_util.visualize_boxes_and_labels_on_image_array(
in run_inference_for_single_image(model, image)
8 # Run inference
9 model_fn = model.signatures['serving_default']
---> 10 output_dict = model_fn(input_tensor)
11
12 # All outputs are batches tensors.
~/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py in call(self, *args, **kwargs)
1603 TypeError: For invalid positional/keyword argument combinations.
1604 """
-> 1605 return self._call_impl(args, kwargs)
1606
1607 def _call_impl(self, args, kwargs, cancellation_manager=None):
~/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _call_impl(self, args, kwargs, cancellation_manager)
1643 raise TypeError("Keyword arguments {} unknown. Expected {}.".format(
1644 list(kwargs.keys()), list(self._arg_keywords)))
-> 1645 return self._call_flat(args, self.captured_inputs, cancellation_manager)
1646
1647 def _filtered_call(self, args, kwargs):
~/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1743 and executing_eagerly):
1744 # No tape is watching; skip to running the function.
-> 1745 return self._build_call_outputs(self._inference_function.call(
1746 ctx, args, cancellation_manager=cancellation_manager))
1747 forward_backward = self._select_forward_and_backward_functions(
~/.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
591 with _InterpolateFunctionError(self):
592 if cancellation_manager is None:
--> 593 outputs = execute.execute(
594 str(self.signature.name),
595 num_outputs=self._num_outputs,
~/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
57 try:
58 ctx.ensure_initialized()
---> 59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
60 inputs, attrs, num_outputs)
61 except core._NotOkStatusException as e:
UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/BatchNorm/batchnorm/mul_1 (defined at :11) ]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/BatchNorm/batchnorm/mul_1 (defined at :11) ]]
[[Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Identity/_43]]
0 successful operations.
0 derived errors ignored. [Op:__inference_pruned_17182]
Function call stack:
pruned -> pruned
2020-10-25 10:58:27.887776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-10-25 10:58:27.932095: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:27.932441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:06:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.695GHz coreCount: 14 deviceMemorySize: 3.82GiB deviceMemoryBandwidth: 119.24GiB/s
2020-10-25 10:58:27.934866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-25 10:58:27.978859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-25 10:58:28.004490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-25 10:58:28.011528: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-25 10:58:28.057270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-25 10:58:28.064023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-25 10:58:28.145338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-25 10:58:28.145777: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.146731: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.147446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-10-25 10:58:28.148134: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-25 10:58:28.160140: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3393235000 Hz
2020-10-25 10:58:28.160818: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fba14000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-25 10:58:28.160864: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-10-25 10:58:28.256111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.256611: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e5fa6c3a10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-25 10:58:28.256626: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1650, Compute Capability 7.5
2020-10-25 10:58:28.257194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.257478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:06:00.0 name: GeForce GTX 1650 computeCapability: 7.5
coreClock: 1.695GHz coreCount: 14 deviceMemorySize: 3.82GiB deviceMemoryBandwidth: 119.24GiB/s
2020-10-25 10:58:28.257520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-25 10:58:28.257534: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-10-25 10:58:28.257545: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-10-25 10:58:28.257557: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-10-25 10:58:28.257567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-10-25 10:58:28.257578: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-10-25 10:58:28.257589: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-25 10:58:28.257655: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.257978: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.258241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-10-25 10:58:28.258853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-10-25 10:58:28.260062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-25 10:58:28.260074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-10-25 10:58:28.260080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-10-25 10:58:28.260730: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.261065: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-25 10:58:28.261355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3357 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650, pci bus id: 0000:06:00.0, compute capability: 7.5)
2020-10-25 10:59:03.502628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-25 10:59:04.206827: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-10-25 10:59:04.215675: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.