Regarding object 3D vertices in meters (8 corners and 1 centroid) extraction and project them to 2D image plane (pixel cordinate)
- Dominant language
- Python
- Stars
- 3.7k
- Forks
- 518
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the issue
I was using a code from [visp](https://github.com/lagadic/visp/blob/master/script/dataset_generator/generate_scene.py). They have a script where you can generate all the images (RGB, depth, normal etc) and a 2D bounding box for yolov7 export. Now there was 1 problem that I saw in their script that the pose estimation w.r.t camera frame seems wrong (cTo) but there bounding box was correct.
Now I want to extract the 8 corners and 1 centroid in 3D space as well as the 2D projection of those 9 vertices on camera plane (9 pixel cordinates). Is there a script you suggest that is already doing it the way visp is doing it (scriptable way) so that I can just incorporate the code block ??
### Minimal code example
They have a bounding box code like this:
```python
def bounding_box_2d_from_vertices(object: bproc.types.MeshObject, K: np.ndarray, camTworld: np.ndarray) -> Tuple[List[float], float, np.ndarray]:
'''
Compute the 2D bounding from an object's vertices
returns A tuple containing:
[xmin, ymin, xmax, ymax] in pixels
the proportion of visible vertices (that are not behind the camera, i.e., negative Z)
The 2D points, in normalized units in camera space, in ViSP/OpenCV frame
'''
worldTobj = homogeneous_no_scaling(object)
obj = object.blender_obj
verts = np.ones(len(obj.data.vertices) * 3)
obj.data.vertices.foreach_get("co", verts)
points = verts.reshape(-1, 3)
# verts = [v.co for v in obj.data.vertices]
camTobj = camTworld @ worldTobj
points_cam = camTobj @ np.concatenate((points, np.ones((len(points), 1))), axis=-1).T
points_cam = convert_points_to_visp_frame((points_cam[:3] / points_cam[3, None]).T)
visible_points = points_cam[points_cam[:, 2] > 0]
visible_points_m_2d = visible_points[:, :2] / visible_points[:, 2, None]
visible_points_px_2d = K @ np.concatenate((visible_points_m_2d, np.ones((len(visible_points_m_2d), 1))), axis=-1).T
visible_points_px_2d = visible_points_px_2d.T[:, :2] / visible_points_px_2d.T[:, 2, None]
mins = np.min(visible_points_px_2d, axis=0)
assert len(mins) == 2
maxes = np.max(visible_points_px_2d, axis=0)
return [mins[0], mins[1], maxes[0], maxes[1]], len(visible_points) / len(points_cam), visible_points_m_2d
```
I tried to add something similar to this:
```python
def project_bounding_box_to_image_plane(object, K, camTworld):
"""
Project the 3D bounding box and centroid of an object onto the 2D image plane using BlenderProc.
Parameters:
- object: bproc.types.MeshObject, the object whose bounding box to project.
- K: The camera intrinsic matrix.
- camTworld: The camera-to-world transformation matrix.
Returns:
- List of 2D coordinates of the projected bounding box vertices and the centroid on the image plane.
"""
# Get the 8 corners of the bounding box in world coordinates
bounding_box_world = np.array(object.get_bound_box())
# Calculate the centroid of the bounding box
centroid_world = np.mean(bounding_box_world, axis=0)
# Combine bounding box corners with the centroid
points_world = np.vstack([bounding_box_world, centroid_world])
# Transform points from world to camera coordinates
worldTcam = np.linalg.inv(camTworld)
points_world_homogeneous = np.hstack((points_world, np.ones((points_world.shape[0], 1))))
points_camera_homogeneous = worldTcam @ points_world_homogeneous.T
# Project points onto image plane
points_image_homogeneous = K @ points_camera_homogeneous[:3, :]
points_image = points_image_homogeneous[:2, :] / points_image_homogeneous[2, :]
return points_image.T.tolist()
```
But it seems that both the object dimension and keypoints didn't match.
### Files required to run the code
_No response_
### Expected behavior
Here was the output:

Both the keypoint location and the dimension is incorrect.
### BlenderProc version
v2.7.0
Contributor guide
Assessment
This issue has not been assessed yet.