isl-org / isl-org/Open3D

Floating-point equality check in RANSAC result comparison may never trigger

Open
#7,256 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [x] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [x] For Python issues, I have tested with the [latest development wheel](https://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [x] I have checked the [release documentation](https://www.open3d.org/docs/release/) and the [latest documentation](https://www.open3d.org/docs/latest/) (for `main` branch).

### Describe the issue

In [PointClooud::SegmentPlane](https://github.com/isl-org/Open3D/blob/main/cpp/open3d/geometry/PointCloudSegmentation.cpp#L214), the RANSAC result comparison includes a direct equality check on fitness_, which is a double:
```
if (this_result.fitness_ > result.fitness_ ||
(this_result.fitness_ == result.fitness_ &&
this_result.inlier_rmse_ < result.inlier_rmse_)) {

```
Since `fitness_` is [defined](https://github.com/isl-org/Open3D/blob/main/cpp/open3d/geometry/PointCloudSegmentation.cpp#L61) as a double, exact equality is highly unlikely due to precision limitations. This makes the tie-breaking condition based on inlier_rmse_ effectively unreachable in most cases.

Suggestion:

```
const double epsilon = 1e-8;
if (this_result.fitness_ > result.fitness_ ||
(std::abs(this_result.fitness_ - result.fitness_) < epsilon &&
this_result.inlier_rmse_ < result.inlier_rmse_)) {
```
This would allow more robust tie-breaking when fitness_ values are nearly equal.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in cpp/open3d/geometry/PointCloudSegmentation.cpp at PointCloud::SegmentPlane and inspect how fitness_ and inlier_rmse_ are compared in the RANSAC result selection. Review the surrounding result calculations before choosing a tolerance, then verify that nearly equal fitness values are tie-broken by inlier_rmse_ without changing the preferred result for clearly higher fitness.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
computer-vision
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.