capture_screen_float_buffer() not affected by arbitrary camera intrinsic
- Dominant language
- C++
- Stars
- 14k
- Forks
- 2.6k
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 6
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).
### Describe the issue
Rendering image using `Visualizer.capture_screen_float_buffer()` seems to be not affected by arbitrary camera intrinsic, which is similar to this [issue](https://github.com/isl-org/Open3D/issues/5049) and could be due to this [issue](https://github.com/pablospe/render_depthmap_example/issues/4)
### Steps to reproduce the bug
```python
import numpy as np
import open3d as o3d
import matplotlib.pyplot as plt
# Define image width and height
w, h = 200, 200
# Define camera intrinsic
fx, fy = 200, 200
px, py = 100, 100
# Create mesh object
mesh = o3d.geometry.TriangleMesh.create_box()
mesh.paint_uniform_color([0,1,0]) # Green
mesh.compute_vertex_normals()
mesh.translate([0,0,5]) # Translate infront of camera
########################################
### Render image with old Visualizer ###
########################################
vis = o3d.visualization.Visualizer()
vis.create_window(width=w, height=h)
vis.add_geometry(mesh)
# Update camera view
cam = o3d.camera.PinholeCameraParameters()
cam.extrinsic = np.eye(4)
cam.intrinsic = o3d.camera.PinholeCameraIntrinsic(w, h, fx, fy, px, py)
ctr = vis.get_view_control()
ctr.convert_from_pinhole_camera_parameters(cam, allow_arbitrary=True) # https://github.com/isl-org/Open3D/pull/2564
# Render image
image_old = np.asarray(vis.capture_screen_float_buffer(do_render=True))
image_old = (image_old * 255).astype(np.uint8) # Scale from [0,1] to [0,255]
###############################################
### Render image with new OffscreenRenderer ###
###############################################
render = o3d.visualization.rendering.OffscreenRenderer(width=w, height=h)
material = o3d.visualization.rendering.MaterialRecord() # Create material
render.scene.add_geometry('mesh', mesh, material)
render.setup_camera(cam.intrinsic, cam.extrinsic)
# Render image
image_new = np.asarray(render.render_to_image())
# Plot image for comparison
fig, (ax1, ax2) = plt.subplots(1,2)
fig.suptitle(f'fx:{fx} fy:{fy} px:{px} py:{py}')
ax1.set_title('Visualizer')
ax2.set_title('OffscreenRenderer')
ax1.set_yticks(np.arange(0,h,50))
ax2.set_yticks(np.arange(0,h,50))
ax1.imshow(image_old)
ax2.imshow(image_new)
ax1.grid()
ax2.grid()
plt.show()
```
### Error message
_No response_
### Expected behavior
The rendered image using `Visualizer.capture_screen_float_buffer()` should match `OffscreenRenderer.render_to_image()`
### Open3D, Python and System information
```markdown
- Operating system: Ubuntu 18.04
- Python version: Python 3.7
- Open3D version: 0.15.2
- Is this a remote workstation?: no
- How did you install Open3D?: pip
```
### Additional information
Ideal scenario where camera intrinsics: fx = fy and px, py are half of image width and height, the image is rendered correctly (top left corner of the green box is rendered at pixel location 100,100)

Changing the py to a different value, the image rendered by Visualizer remains the same, the expected result should be OffscreenRenderer

Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.