Floating-point equality check in RANSAC result comparison may never trigger
- Dominant language
- C++
- Stars
- 14k
- Forks
- 2.6k
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 6
Description
### Checklist
- [x] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [x] For Python issues, I have tested with the [latest development wheel](https://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [x] I have checked the [release documentation](https://www.open3d.org/docs/release/) and the [latest documentation](https://www.open3d.org/docs/latest/) (for `main` branch).
### Describe the issue
In [PointClooud::SegmentPlane](https://github.com/isl-org/Open3D/blob/main/cpp/open3d/geometry/PointCloudSegmentation.cpp#L214), the RANSAC result comparison includes a direct equality check on fitness_, which is a double:
```
if (this_result.fitness_ > result.fitness_ ||
(this_result.fitness_ == result.fitness_ &&
this_result.inlier_rmse_ < result.inlier_rmse_)) {
```
Since `fitness_` is [defined](https://github.com/isl-org/Open3D/blob/main/cpp/open3d/geometry/PointCloudSegmentation.cpp#L61) as a double, exact equality is highly unlikely due to precision limitations. This makes the tie-breaking condition based on inlier_rmse_ effectively unreachable in most cases.
Suggestion:
```
const double epsilon = 1e-8;
if (this_result.fitness_ > result.fitness_ ||
(std::abs(this_result.fitness_ - result.fitness_) < epsilon &&
this_result.inlier_rmse_ < result.inlier_rmse_)) {
```
This would allow more robust tie-breaking when fitness_ values are nearly equal.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in cpp/open3d/geometry/PointCloudSegmentation.cpp at PointCloud::SegmentPlane and inspect how fitness_ and inlier_rmse_ are compared in the RANSAC result selection. Review the surrounding result calculations before choosing a tolerance, then verify that nearly equal fitness values are tie-broken by inlier_rmse_ without changing the preferred result for clearly higher fitness.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100