How to extend the registration pipelines
- Dominant language
- C++
- Stars
- 14k
- Forks
- 2.6k
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 6
Description
### Checklist
- [x] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [x] For Python issues, I have tested with the [latest development wheel](https://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [x] I have checked the [release documentation](https://www.open3d.org/docs/release/) and the [latest documentation](https://www.open3d.org/docs/latest/) (for `main` branch).
### My Question
Hi,
I worked around this issue for a while, but I think I want to give it a try. I would love to be able to _customize_ the registration pipelines in the open3d library (which I think are amazing and a very good point for using this library over others).
# Idea
I want to change the registration so that it can be constrained (or the objective can be extended, but this could be a later step).
Personally, I am working with python and I am less proficient in C++, so I would love to be able to do this through the python bindings API without changing and writing C++ code. Once everything works, we can see how to translate/port it. I am willing to work on this.
## Long Version
### Background/Issues Check
I am also aware of these issues:
- [ [Feature request] Registration with constraints #409](https://github.com/isl-org/Open3D/issues/409)
This is the most similar. The registration by constraining the rotation. Although the end of that discussion is that there is one implementation ([this commit](https://github.com/grossjohannes/Open3D/commit/423490c92095352fd412048d005e04ae4f15308a) from [this fork](https://github.com/grossjohannes/Open3D)), it seems limited to setting the z-axis to zero, and it would be nice to be able to have some freedom on the constraints.
- [How to set contraint on transformation matrix? Issue #5935](https://github.com/isl-org/Open3D/issues/5935)
This one is about stitching point cloud, so it is still a similar problem (constrained transformation), but did not reach any point
- [Activate ICP on a subset of points based on a condition on angle between local normals of two point clouds for ICP, Issue #3139](https://github.com/isl-org/Open3D/issues/3139)
This is also interesting, as it would need a different constraint, pushing for a more flexible solution (not just setting some values to 0)
- [How to stitch point clouds with no overlap? Issue #6995](https://github.com/isl-org/Open3D/discussions/6995)
Again, in this case with static sensor it was solved, but could be seen as part of the same problem.
### Question
So, to my question. Mostly, I thought about how to do it, and wanted to understand if I am in the right direction.
Checking the documentation, I see that a registration (for example, [registration_icp](https://www.open3d.org/docs/latest/python_api/open3d.pipelines.registration.registration_icp.html#open3d.pipelines.registration.registration_icp)) relies on a [TransformationEstimation](https://www.open3d.org/docs/latest/python_api/open3d.pipelines.registration.TransformationEstimation.html#open3d.pipelines.registration.TransformationEstimation), which could be extended/customized. The [TransformationEstimation](https://www.open3d.org/docs/latest/python_api/open3d.pipelines.registration.TransformationEstimation.html#open3d.pipelines.registration.TransformationEstimation) only needs to implement 3 methods (`__init__`, `compute_rmse` and `compute_transformation`) as far as I understand, so it seems doable.
If I would just create a `ConstrainedTransformationEstimation` which accepts one axis as a parameter, and just ignore that axis (when doing `compute_transformation`, just use the other two axis?), would that be enough to actually perform a 2D registration on a pointcloud?
Should I use the [pipelines](https://www.open3d.org/docs/latest/python_example/pipelines/index.html) as a reference, for example the [`doppler-icp-registration.py`](https://www.open3d.org/docs/latest/python_example/pipelines/index.html#doppler-icp-registration-py)?
I am a bit afraid that the correspondences may cause some issues if the correct transformation would need the third dimension. But to me changing the TransformationEstimation seems the way to go (as it is changed between the _standard_ ICP and the _colored_ ICP for example [in the tutorial](https://www.open3d.org/docs/latest/tutorial/t_pipelines/t_icp_registration.html#ICP-registration)).
Apart from this, I am not understanding where exactly the `correspondences` (point 1 of [Understanding-ICP-Algorithm](https://www.open3d.org/docs/latest/tutorial/t_pipelines/t_icp_registration.html#Understanding-ICP-Algorithm)) are calculated (in each iteration of the ICP), could you point me out (so that can also be changed).
## TL;DR
1. If I would just create a `ConstrainedTransformationEstimation` (extending TransformationEstimation) which accepts one axis as a parameter, and just ignore that axis (when doing `compute_transformation`, just use the other two axis?), would that be enough to actually perform a 2D registration on a pointcloud?
Should I use the [pipelines](https://www.open3d.org/docs/latest/python_example/pipelines/index.html) as a reference, for example the [`doppler-icp-registration.py`](https://www.open3d.org/docs/latest/python_example/pipelines/index.html#doppler-icp-registration-py)?
2. could you point me out where exactly the `correspondences` are calculated (in each iteration of the ICP)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.