For the o3d.visualization.O3DVisualizer class, how to add another panel or window with image widgets and other info?
- Dominant language
- C++
- Stars
- 14k
- Forks
- 2.6k
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 6
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](https://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](https://www.open3d.org/docs/release/) and the [latest documentation](https://www.open3d.org/docs/latest/) (for `main` branch).
### My Question
Hi there, thanks for the great library!
### Context
I am trying to add a gui to the mapping script of the [concept-graphs mapping system](https://github.com/concept-graphs/concept-graphs/tree/c52043b177ee10816dfd9a1509e2ee746ae459b7). The code is ugly at the moment but you can have a look [here](https://github.com/concept-graphs/concept-graphs/blob/c52043b177ee10816dfd9a1509e2ee746ae459b7/conceptgraph/slam/gui_realtime_mapping.py).
This mapping system takes as input rgb images + depth images + poses, and iteratively constructs an object based 3D map that also encodes semantic features. I want to be able to visualize the whole map building process, so I can inspect the map and debug things as it's being built.
So I looked at the python examples and saw that the [multiple_windows.py](https://www.open3d.org/docs/release/python_example/visualization/index.html#multiple-windows-py) example seems to have what I want, which is to visualize the object pointclouds and the bounding boxes and so on. I can do that, it works great so far.

### The Problem
The issue is that I want to also display more information. I would like to be able to see in a panel beside the map: the current rgb image, the current depth image, the current annotated image, the current frame, the current number of objects and so on.
I want it to be like the [dense_slam_gui.py](https://github.com/isl-org/Open3D/blob/main/examples/python/t_reconstruction_system/dense_slam_gui.py) example in the docs.
I have tried a bunch of stuff but I cannot seem to be able to add a panel to the o3d.visualization.O3DVisualizer thing, and I was also unable to add another window or something that I can update at the same time. I don't mind if its not the same window, I just want to be able to see this information displayed somewhere alongside the map, as it would help me improve the mapping system.
### My Code
Here are some relevant code snippets.
How I make the app (as you can see I have tried stuff to add another window or panel, but everything I have tried so far has resulted in some kind of error)
```python
def run(self):
app = o3d.visualization.gui.Application.instance
app.initialize()
self.main_vis = o3d.visualization.O3DVisualizer(
"Open3D - Multi-Window Demo")
self.main_vis.add_action("Take snapshot in new window",
self.on_snapshot)
self.main_vis.add_action("Pause/Resume updates", lambda vis: self.toggle_pause())
self.main_vis.set_on_close(self.on_main_window_closing)
app.add_window(self.main_vis)
# app.add_window(self.images_window)
# # Setup the secondary window for images
# # self.setup_image_display_window()
# # self.image_window = app.create_window("RGB and Depth Images", 640, 480)
# self.image_window = gui.Application.instance.create_window("RGB and Depth Images", 640, 480)
# # self.image_window.create_window()
# # self.image_window = o3d.visualization.Visualizer("RGB and Depth Images", 640, 480)
# self.layout = o3d.visualization.gui.Vert(0, o3d.visualization.gui.Margins(10))
# self.image_window.add_child(self.layout)
# # Create image widgets
# self.rgb_widget = o3d.visualization.gui.ImageWidget()
# self.depth_widget = o3d.visualization.gui.ImageWidget()
# # Add image widgets to the layout
# self.layout.add_child(self.rgb_widget)
# self.layout.add_child(self.depth_widget)
# app.add_window(self.image_window)
self.snapshot_pos = (self.main_vis.os_frame.x, self.main_vis.os_frame.y)
threading.Thread(target=self.update_thread).start()
app.run()
```
Here is how I do the update thread:
```python
def update_thread(self):
# This is NOT the UI thread, need to call post_to_main_thread() to update
# the scene or any part of the UI.
... lots of code, basically the whole mapping script ...
def my_update_cloud():
# Remove previous objects
for obj_name in self.prev_obj_names:
self.main_vis.remove_geometry(obj_name)
self.prev_obj_names = []
# Remove previous bounding boxes
for bbox_name in self.prev_bbox_names:
self.main_vis.remove_geometry(bbox_name)
self.prev_bbox_names = []
# Add the new objects and bounding boxes
for obj_num, obj in enumerate(self.objects):
obj_label = f"{obj['curr_obj_num']}_{obj['class_name']}"
obj_name = f"obj_{obj_label}"
bbox_name = f"bbox_{obj_label}"
self.prev_obj_names.append(obj_name)
self.main_vis.add_geometry(obj_name, obj['pcd'])
self.prev_bbox_names.append(bbox_name)
self.main_vis.add_geometry(bbox_name, obj['bbox'] )
if self.is_done: # might have changed while sleeping
break
o3d.visualization.gui.Application.instance.post_to_main_thread(
self.main_vis, my_update_cloud)
```
### What I want to do
For starters, I just want to be able to update a display with the current rgb and depth image and maybe the current frame number?. I'm imagining something like:
```python
def update_images(self):
# Convert numpy images to Open3D images and update widgets
o3d_rgb_image = o3d.geometry.Image(self.curr_image_rgb)
o3d_depth_image = o3d.geometry.Image((self.curr_depth_array / self.curr_depth_array.max() * 255).astype(np.uint8)) # Example normalization
self.rgb_widget.update_image(o3d_rgb_image)
self.depth_widget.update_image(o3d_depth_image)
# and then in the main update thread loop I would do
self.curr_image_rgb = image_rgb
self.curr_depth_array = depth_array
o3d.visualization.gui.Application.instance.post_to_main_thread(
self.image_window, self.update_images())
```
I hope it makes sense, what I'm trying to achieve here. If anyone could point me in the right direction, this would be much appreciated. I have googled and asked chatGPT a lot to no avail.
Thank you in advance for your time and help.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.