CUDA "driver shutting down" Error when built with shared libraries
- Dominant language
- C++
- Stars
- 14k
- Forks
- 2.6k
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 6
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).
### Describe the issue
When Open3D is linked as a dynamic library, creating a Open3d::core::Tensor object using the default constructor and then reassigning the object to a new instance with a new data allocation results in the following error when the application is closed:
```
[Open3D Error] (void __cdecl open3d::core::__OPEN3D_CUDA_CHECK(enum cudaError,const char *,const int)) C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:289: C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:114 CUDA runtime error: driver shutting down
```
This causes the application to hang for several seconds likely due to a crash on exit
This issue does not occur when the same code is run with Open3D built and linked as a static library.
[Repository with minimal CMake project to reproduce](https://github.com/MimetrikDE/Open3D_Tensor_Assign_Issue)
### Steps to reproduce the bug
```
1. Build Open3D with shared libraries as described below
2. Create a Open3D::core::Tensor object using the default consructor
3. Pass the Tensor object to a function by reference
4. Assign a new Tensor instance to the passed object with a allocation on the device ("CUDA:0")
5. Wait for program to exit
6. Application crashes before closing with the stated error message.
#include
#include
void AssignNew(open3d::core::Tensor& testTensor) {
testTensor = open3d::core::Tensor::Zeros({ 100,3 }, open3d::core::Dtype::Float32, open3d::core::Device("CUDA:0"));
}
int main() {
std::cout << "Start" << std::endl;
open3d::core::Tensor testTensor;
AssignNew(testTensor);
open3d::core::cuda::ReleaseCache();
std::cout << "Finished program" << std::endl;
return 0;
}
```
### Error message
```
Start
Finished program
[Open3D Error] (void __cdecl open3d::core::__OPEN3D_CUDA_CHECK(enum cudaError,const char *,const int)) C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:289: C:\Users\User\Repositories\Open3D\cpp\open3d\core\CUDAUtils.cpp:114 CUDA runtime error: driver shutting down
```
### Expected behavior
For the given example code and CMake project I expect the application to replace the empty Tensor instance passed into the function with a new Tensor instance holding a device allocation and exit the program cleanly.
### Open3D, Python and System information
```markdown
## Open3D Build Flags
- BUILD_CUDA_MODULE : True
- BUILD_SHARED_LIBS : True
- BUILD_WEBRTC : False
- STATIC_WINDOWS_RUNTIME : False
- BUILD_PYTHON_MODULE : False
Built using
- Visual Studio 17 2022
- CMake 3.25.1
- Windows 10 64-bit x86
- C++17
- Open3D Version: 0.17.0, Commit: 5b6ef4b04b1a4184f12a2c1181ad2b7d2fe45248
System information
- i9 12900H
- RTX 3080 (Laptop)
- 16GB DDR4 RAM
```
### Additional information
None
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.