isl-org / isl-org/Open3D

open3d point cloud rendering doesn't match opencv 2D projection

Open
#6,208 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### My Question

I have a point cloud already transformed into the camera coordinate frame so the projection matrix should be pretty much the same as the intrinsics . My attempts to render the point cloud does not match my projection written in OpenCV (as the point cloud is already aligned to the camera coordinate frame, I can project it into 2D directly using values from the camera instrinsic matrix in OpenCV).

My code for rendering the point cloud `pcl` via Open3D is as follows:
```
img_width = 640
img_height = 512

render = o3d.visualization.rendering.OffscreenRenderer(width=img_width, height=img_height)
render.scene.set_background([0.0, 0.0, 0.0, 1.0]) # RGBA

mtl = o3d.visualization.rendering.MaterialRecord()
mtl.base_color = [1.0, 1.0, 1.0, 1.0] # RGBA
mtl.shader = "defaultUnlit"

render.scene.add_geometry("model", pcl, mtl)

extrinsics_matrix = np.eye(4)
render.setup_camera(intrinsics_matrix, extrinsics_matrix, img_width, img_height)

center = [0, 0, 1]
eye = [0, 0, 0]
up = [0, 1, 0]
render.scene.camera.look_at(center, eye, up)

output = render.render_to_image()
```

Open3D rendering:

vs OpenCV projection:

There seems to be translation and possibly scale differences between the two outputs.

Lastly, I'm confused what the difference is between `render.setup_camera`, `render.scene.camera.set_projection`, and `render.scene.camera.look_at`.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.