isl-org / isl-org/Open3D-ML

RuntimeError: CUDA error: an illegal memory access was encountered

Open
#650 1 comment 1 reaction 0 assignees View on GitHub
question
Dominant language
Python
Stars
2.3k
Forks
365
Avg merge
4h 6m
Merged PRs (30d)
1

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D-ML/issues).
- [X] I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `main` branch).

### My Question

running the following script
(o3dml) abhnegi@tghfg~/o3dml/Open3D-ML$ python scripts/run_pipeline.py torch -c ml3d/configs/pointpillars_kitti.yml --split test --dataset.dataset_path "/pfs/rdi/cea/rdicea_vru/01_Datasets/Kitti/" --pipeline ObjectDetection --dataset.use_cache True

gives me this error :
Traceback (most recent call last):
File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 261, in
sys.exit(main())
File "/pfs/rdi/cea/home/abhnegi/o3dml/Open3D-ML/scripts/run_pipeline.py", line 190, in main
pipeline.run_test()
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 114, in run_test
self.run_valid()
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 191, in run_valid
results = model(data)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 132, in forward
x = self.extract_feats(inputs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 104, in extract_feats
voxels, num_points, coors = self.voxelize(points)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 117, in voxelize
res_voxels, res_coors, res_num_points = self.voxel_layer(res)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/abhnegi/miniconda3/envs/o3dml/lib/python3.10/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 361, in forward
[torch.zeros_like(points_feats[0:1, :]), points_feats])
RuntimeError: CUDA error: an illegal memory access was encountered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.