facebookresearch / facebookresearch/sam-3d-objects

Incorrect Alignment When Rendering Views with Custom FOV and Camera Radius

Open
#100 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.4k
Forks
878
PR merge metrics
No merged PRs in 30d

Description

Hi, thanks for the great work on SAM3D-Object!
I encountered an issue when trying to reproduce multi-view consistency using my own Blender-rendered images.

🐛 Issue Description

I rendered an object in Blender with:

FOV = 49.1

Camera radius = 1.5

Views: azimuths = 0°, 90°, 180°, 270°, elevation = 0°

Then:

I used the azimuth = 0° image as input to sam3d-object to generate the 3D reconstruction.

I attempted to render the generated object from azimuths 90°, 180°, 270° using the notebook’s default settings:

FOV = 60

r = 1

→ The rendered views do not align with my original Blender images.

I also tried rendering with my original parameters (FOV = 49.1, r = 1.5) inside the SAM3D-Object rendering pipeline.

→ The output still fails to align with the original rendered images.

Image

Image

The rendered results is too big compare to the gt input and render images

🔍 Rendering Code Used
# yaw = azimuth
yaws = [math.radians(a + azi_bias) for a in azimuth_list]

# pitch = elevation
pitchs = [math.radians(elevation_deg)] * len(azimuth_list)

extr, intr = _yaw_pitch_r_fov_to_extrinsics_intrinsics(yaws, pitchs, r, fov)

out = render_utils.render_frames(
sample,
extr,
intr,
{"resolution": resolution, "bg_color": bg_color, "backend": "gsplat"},
**kwargs,
)
return out # out["color"] shape = [N, H, W, 3]

And the inference workflow:

from utils.inference import Inference, render_video, make_scene

outputs = [self.inference(image_rgb, mask, seed=seed) for mask in masks]

# 5. Merge scene and prepare for rendering
scene_gs = make_scene(*outputs)
scene_gs_ready = ready_gaussian_for_video_rendering(scene_gs)

❓ Question

Is there a known mismatch between Blender's camera model and the _yaw_pitch_r_fov_to_extrinsics_intrinsics conversion used in SAM3D-Object?

Or is there an additional transformation needed to ensure the rendered views match Blender’s coordinate conventions?

Any guidance on how to correctly align the camera parameters would be greatly appreciated!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.