isl-org / isl-org/Open3D

CUDA "driver shutting down" Error when built with shared libraries

Open
#6,399 1 comment 1 reaction 0 assignees View on GitHub
bug
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### Describe the issue

When Open3D is linked as a dynamic library, creating a Open3d::core::Tensor object using the default constructor and then reassigning the object to a new instance with a new data allocation results in the following error when the application is closed:

```
[Open3D Error] (void __cdecl open3d::core::__OPEN3D_CUDA_CHECK(enum cudaError,const char *,const int)) C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:289: C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:114 CUDA runtime error: driver shutting down
```
This causes the application to hang for several seconds likely due to a crash on exit

This issue does not occur when the same code is run with Open3D built and linked as a static library.

[Repository with minimal CMake project to reproduce](https://github.com/MimetrikDE/Open3D_Tensor_Assign_Issue)

### Steps to reproduce the bug

```
1. Build Open3D with shared libraries as described below
2. Create a Open3D::core::Tensor object using the default consructor
3. Pass the Tensor object to a function by reference
4. Assign a new Tensor instance to the passed object with a allocation on the device ("CUDA:0")
5. Wait for program to exit
6. Application crashes before closing with the stated error message.


#include
#include

void AssignNew(open3d::core::Tensor& testTensor) {
testTensor = open3d::core::Tensor::Zeros({ 100,3 }, open3d::core::Dtype::Float32, open3d::core::Device("CUDA:0"));
}

int main() {

std::cout << "Start" << std::endl;

open3d::core::Tensor testTensor;
AssignNew(testTensor);

open3d::core::cuda::ReleaseCache();

std::cout << "Finished program" << std::endl;

return 0;
}
```

### Error message
```
Start
Finished program
[Open3D Error] (void __cdecl open3d::core::__OPEN3D_CUDA_CHECK(enum cudaError,const char *,const int)) C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:289: C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:114 CUDA runtime error: driver shutting down
```

### Expected behavior

For the given example code and CMake project I expect the application to replace the empty Tensor instance passed into the function with a new Tensor instance holding a device allocation and exit the program cleanly.

### Open3D, Python and System information

```markdown
## Open3D Build Flags
- BUILD_CUDA_MODULE : True
- BUILD_SHARED_LIBS : True
- BUILD_WEBRTC : False
- STATIC_WINDOWS_RUNTIME : False
- BUILD_PYTHON_MODULE : False

Built using
- Visual Studio 17 2022
- CMake 3.25.1
- Windows 10 64-bit x86
- C++17
- Open3D Version: 0.17.0, Commit: 5b6ef4b04b1a4184f12a2c1181ad2b7d2fe45248

System information
- i9 12900H
- RTX 3080 (Laptop)
- 16GB DDR4 RAM
```

### Additional information

None

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.