PointCloudLibrary / PointCloudLibrary/pcl
[IterativeClosestPoint] Viewpoint information not handled during registration
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 11.1k
- Forks
- 4.7k
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 6
Description
Describe the bug
Registration<PointSource, PointTarget, Scalar>::align(PointCloudSource& output) return wrong registration result when the input 2 point cloud have different viewpoint.
Context
Registration<PointSource, PointTarget, Scalar>::align(PointCloudSource& output) does not take into account the viewpoint differences between the two input point clouds. The returned point cloud's data field is usually consistent with the reference cloud. However, differing viewpoints can lead to significant positional discrepancies when visualized using pcl::visualizer, since the visualizer considers the transformation from the camera coordinate system to the world coordinate system based on the viewpoint, whereas icp.align() does not.
Expected behavior
(though not perfectly matched,it seems like a right behaviour)
Current Behavior
To Reproduce
I provide a simple example to reproduce my question
#include <iostream>
#include <fstream>
#include <thread>
#include <pcl/io/ply_io.h>
#include <pcl/io/pcd_io.h>
#include <pcl/filters/voxel_grid.h>
#include <pcl/registration/icp.h>
#include <pcl/common/transforms.h>
#include <pcl/visualization/pcl_visualizer.h>
int main() {
std::string sFileName_mov = R"(E:\bunny.pcd)";
std::string sFileName_ref = R"(E:\bunnyT.pcd)";
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud_mov(new pcl::PointCloud<pcl::PointXYZ>);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud_ref(new pcl::PointCloud<pcl::PointXYZ>);
if (pcl::io::loadPCDFile(sFileName_mov, *cloud_mov) < 0 ||
pcl::io::loadPCDFile(sFileName_ref, *cloud_ref) < 0) {
std::cerr << "Failed to load point clouds." << std::endl;
return -1;
}
// if set viewpoint to zero, program run as expected
//cloud_mov->sensor_origin_.setZero();
//cloud_mov->sensor_orientation_ = Eigen::Quaternionf::Identity();
//cloud_ref->sensor_origin_.setZero();
//cloud_ref->sensor_orientation_ = Eigen::Quaternionf::Identity();
pcl::IterativeClosestPoint<pcl::PointXYZ, pcl::PointXYZ> icp;
icp.setInputSource(cloud_mov);
icp.setInputTarget(cloud_ref);
icp.setMaximumIterations(100);
icp.setMaxCorrespondenceDistance(0.05);
icp.setTransformationEpsilon(1e-8);
icp.setEuclideanFitnessEpsilon(1e-6);
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud_icp_result(new pcl::PointCloud<pcl::PointXYZ>);
icp.align(*cloud_icp_result);
if (icp.hasConverged()) {
std::cout << "ICP converged. Score: " << icp.getFitnessScore() << std::endl;
}
else {
std::cerr << "ICP did not converge." << std::endl;
}
// visualization
pcl::visualization::PCLVisualizer::Ptr viewer(new pcl::visualization::PCLVisualizer("ICP Result Viewer"));
viewer->setBackgroundColor(0, 0, 0);
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> ref_color(cloud_ref, 0, 0, 255);
viewer->addPointCloud<pcl::PointXYZ>(cloud_ref, ref_color, "reference cloud");
pcl::visualization::PointCloudColorHandlerCustom<pcl::PointXYZ> mov_color(cloud_icp_result, 255, 0, 0);
viewer->addPointCloud<pcl::PointXYZ>(cloud_icp_result, mov_color, "aligned cloud");
viewer->setPointCloudRenderingProperties(pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 2, "reference cloud");
viewer->setPointCloudRenderingProperties(pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 2, "aligned cloud");
viewer->addCoordinateSystem(0.1);
while (!viewer->wasStopped()) {
viewer->spinOnce(100);
}
return 0;
}
the main part to reproduce the result is the pcd file "bunny.pcd" and "bunnyT.pcd".I cannot directly upload this to github.So I provide google drive link here:https://drive.google.com/file/d/1JxzXIKyi-IC3wjB3d7cJMqlBQNV_3gle/view?usp=drive_link, https://drive.google.com/file/d/1IRVhzo26caWpQKDPQZ2jaR7Y5IrnyKpU/view?usp=drive_link.The key point is that the two PCD files have different "VIEWPOINT" fields. Since they are in ASCII format, you can easily inspect them using any text editor.
Your Environment (please complete the following information):
- OS: Windows 10
- Compiler: MSVC 1940
- PCL Version 1.15.1
Possible Solution
1.When performing point cloud registration, if the viewpoints of the two input clouds are different, a warning should be issued to the user.
2. Transform the data to the world coordinate system for both point clouds before performing the registration.
or align one point cloud's viewpoint with the other by transforming the data of one cloud to match the viewpoint of the other.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with pcl/registration/icp.h and the Registration::align entry point, then inspect how the input clouds' VIEWPOINT metadata is handled alongside pcl/common/transforms.h. Reproduce the discrepancy using the bunny.pcd and bunnyT.pcd files with different VIEWPOINT fields. Done should be a decided and tested behavior for differing viewpoints, such as warning users or consistently transforming the data before registration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100