isl-org / isl-org/Open3D

Out of memory error on Multi-ICP with open3d tensor point clouds

Open
#6,018 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### Describe the issue

I have been running multi ICP in the tensor form and whenever I reach a certain sample it runs to out of memory:

```
RuntimeError: [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /root/Open3D/cpp/open3d/core/CUDAUtils.cpp:289: /root/Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp:24 CUDA runtime error: out of memory

```

I used this example as help: [http://www.open3d.org/docs/release/tutorial/t_pipelines/t_icp_registration.html#Multi-Scale-ICP-Example](http://www.open3d.org/docs/release/tutorial/t_pipelines/t_icp_registration.html#Multi-Scale-ICP-Example)

I have two point clouds loaded as a torch tensor and can correctly create a open3d.t Point cloud and multi-ICP is working with 1.9GB GPU memory occupied but after 100 point cloud pair predictions the memory jumps to 3GB and after the 250th pair it runs out of memory.

I also tried to specificaly `del source_pcd, target_pcd` without help.

I hope you can help me.

### Steps to reproduce the bug

```python
import open3d as o3d

RT_pred = o3d_multi_icp_tensor(source, target)

def o3d_multi_icp_tensor(source: torch.Tensor,
target: torch.Tensor,
trans_init: Union[torch.Tensor, np.ndarray] = torch.eye(4),
voxel_sizes = [0.5, 0.3, 0.1],
corr_distances = [1.5, 0.9, 0.3],
max_iter = [20, 15, 10],
rmse_fit = [0.001, 0.0001, 0.00001]):
if isinstance(trans_init, torch.Tensor):
trans_init = trans_init.cpu().numpy()
voxel_sizes = o3d.utility.DoubleVector(voxel_sizes)
criteria_list = []
for i in range(len(max_iter)):
criteria_list.append(o3d.t.pipelines.registration.ICPConvergenceCriteria(relative_fitness=rmse_fit[i],
relative_rmse=rmse_fit[i],
max_iteration=max_iter[i]))

max_correspondence_distances = o3d.utility.DoubleVector(corr_distances)
init_source_to_target = o3d.core.Tensor.eye(4, o3d.core.Dtype.Float32)
estimation = o3d.t.pipelines.registration.TransformationEstimationPointToPlane()

reg_p2p = o3d.t.pipelines.registration.multi_scale_icp(source, target, voxel_sizes,
criteria_list,
max_correspondence_distances,
init_source_to_target, estimation)

return reg_p2p
```

### Error message

RuntimeError: [Open3D Error] (void open3d::core::__OPEN3D_CUDA_CHECK(cudaError_t, const char*, int)) /root/Open3D/cpp/open3d/core/CUDAUtils.cpp:289: /root/Open3D/cpp/open3d/core/MemoryManagerCUDA.cpp:24 CUDA runtime error: out of memory

### Expected behavior

_No response_

### Open3D, Python and System information

```markdown
- Operating system: Ubuntu 20.04
- Python version: Python 3.9
- Open3D version: 0.17.0 (same problem in 0.16.0)
- System architecture: x86
- Is this a remote workstation?: yes
- How did you install Open3D?: pip
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.