facebookresearch / facebookresearch/detectron2

RuntimeError: CUDA error: invalid argument when running `tracer.export_caffe2()`

Open
#3,998 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34.7k
Forks
7.9k
PR merge metrics
No merged PRs in 30d

Description

## Instructions To Reproduce the Issue:

Run my ['Detectron2: Train and export model.ipynb'](https://colab.research.google.com/github/TannerGilbert/Object-Detection-and-Image-Segmentation-with-Detectron2/blob/master/Detectron2_train_and_export_model.ipynb#scrollTo=3O22vkBFj_6u) notebook.

## Expected behavior:

Detectron2 should let me export my object detector as a caffe2 model as described in the [export documentation](https://detectron2.readthedocs.io/en/latest/modules/export.html), but the following error occurs when running [`export_caffe2()`](https://detectron2.readthedocs.io/en/latest/modules/export.html?highlight=export_caffe2#detectron2.export.Caffe2Tracer.export_caffe2).

Error:
```
RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

## Environment:

Google Colab

Contributor guide

Open the contributing guide

Research direction

Run the linked Detectron2 training and export notebook in Google Colab, then reproduce the failure at tracer.export_caffe2(). Read the export_caffe2 documentation and rerun with CUDA_LAUNCH_BLOCKING=1 to obtain a synchronous traceback. Done means the object detector exports as a Caffe2 model without the CUDA invalid-argument error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.