RuntimeError: CUDA error: an illegal memory access was encountered
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 365
- Avg merge
- 4h 6m
- Merged PRs (30d)
- 1
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `main` branch).
### My Question
running the following script
(o3dml) abhnegi@tghfg~/o3dml/Open3D-ML$ python scripts/run_pipeline.py torch -c ml3d/configs/pointpillars_kitti.yml --split test --dataset.dataset_path "/pfs/rdi/cea/rdicea_vru/01_Datasets/Kitti/" --pipeline ObjectDetection --dataset.use_cache True
gives me this error :
Traceback (most recent call last):
File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 261, in
sys.exit(main())
File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 190, in main
pipeline.run_test()
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 114, in run_test
self.run_valid()
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 191, in run_valid
results = model(data)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 132, in forward
x = self.extract_feats(inputs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 104, in extract_feats
voxels, num_points, coors = self.voxelize(points)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 117, in voxelize
res_voxels, res_coors, res_num_points = self.voxel_layer(res)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 361, in forward
[torch.zeros_like(points_feats[0:1, :]), points_feats])
RuntimeError: CUDA error: an illegal memory access was encountered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.