isl-org / isl-org/Open3D

capture_screen_float_buffer() not affected by arbitrary camera intrinsic

Open
#5,086 3 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### Describe the issue

Rendering image using `Visualizer.capture_screen_float_buffer()` seems to be not affected by arbitrary camera intrinsic, which is similar to this [issue](https://github.com/isl-org/Open3D/issues/5049) and could be due to this [issue](https://github.com/pablospe/render_depthmap_example/issues/4)

### Steps to reproduce the bug

```python
import numpy as np
import open3d as o3d
import matplotlib.pyplot as plt

# Define image width and height
w, h = 200, 200
# Define camera intrinsic
fx, fy = 200, 200
px, py = 100, 100

# Create mesh object
mesh = o3d.geometry.TriangleMesh.create_box()
mesh.paint_uniform_color([0,1,0]) # Green
mesh.compute_vertex_normals()
mesh.translate([0,0,5]) # Translate infront of camera

########################################
### Render image with old Visualizer ###
########################################
vis = o3d.visualization.Visualizer()
vis.create_window(width=w, height=h)
vis.add_geometry(mesh)
# Update camera view
cam = o3d.camera.PinholeCameraParameters()
cam.extrinsic = np.eye(4)
cam.intrinsic = o3d.camera.PinholeCameraIntrinsic(w, h, fx, fy, px, py)
ctr = vis.get_view_control()
ctr.convert_from_pinhole_camera_parameters(cam, allow_arbitrary=True) # https://github.com/isl-org/Open3D/pull/2564
# Render image
image_old = np.asarray(vis.capture_screen_float_buffer(do_render=True))
image_old = (image_old * 255).astype(np.uint8) # Scale from [0,1] to [0,255]

###############################################
### Render image with new OffscreenRenderer ###
###############################################
render = o3d.visualization.rendering.OffscreenRenderer(width=w, height=h)
material = o3d.visualization.rendering.MaterialRecord() # Create material
render.scene.add_geometry('mesh', mesh, material)
render.setup_camera(cam.intrinsic, cam.extrinsic)
# Render image
image_new = np.asarray(render.render_to_image())

# Plot image for comparison
fig, (ax1, ax2) = plt.subplots(1,2)
fig.suptitle(f'fx:{fx} fy:{fy} px:{px} py:{py}')
ax1.set_title('Visualizer')
ax2.set_title('OffscreenRenderer')
ax1.set_yticks(np.arange(0,h,50))
ax2.set_yticks(np.arange(0,h,50))
ax1.imshow(image_old)
ax2.imshow(image_new)
ax1.grid()
ax2.grid()
plt.show()
```

### Error message

_No response_

### Expected behavior

The rendered image using `Visualizer.capture_screen_float_buffer()` should match `OffscreenRenderer.render_to_image()`

### Open3D, Python and System information

```markdown
- Operating system: Ubuntu 18.04
- Python version: Python 3.7
- Open3D version: 0.15.2
- Is this a remote workstation?: no
- How did you install Open3D?: pip
```

### Additional information

Ideal scenario where camera intrinsics: fx = fy and px, py are half of image width and height, the image is rendered correctly (top left corner of the green box is rendered at pixel location 100,100)
![Figure_1](https://user-images.githubusercontent.com/38412160/167334305-1fe0aa90-0ebc-41e0-b986-ae6d6d403bdd.png)

Changing the py to a different value, the image rendered by Visualizer remains the same, the expected result should be OffscreenRenderer
![Figure_2](https://user-images.githubusercontent.com/38412160/167334414-df418abd-c420-4e44-97fc-4efa74ec2069.png)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.