facebookresearch / facebookresearch/sam-3d-objects
Incorrect Alignment When Rendering Views with Custom FOV and Camera Radius
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 878
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thanks for the great work on SAM3D-Object!
I encountered an issue when trying to reproduce multi-view consistency using my own Blender-rendered images.
🐛 Issue Description
I rendered an object in Blender with:
FOV = 49.1
Camera radius = 1.5
Views: azimuths = 0°, 90°, 180°, 270°, elevation = 0°
Then:
I used the azimuth = 0° image as input to sam3d-object to generate the 3D reconstruction.
I attempted to render the generated object from azimuths 90°, 180°, 270° using the notebook’s default settings:
FOV = 60
r = 1
→ The rendered views do not align with my original Blender images.
I also tried rendering with my original parameters (FOV = 49.1, r = 1.5) inside the SAM3D-Object rendering pipeline.
→ The output still fails to align with the original rendered images.
The rendered results is too big compare to the gt input and render images
🔍 Rendering Code Used
# yaw = azimuth
yaws = [math.radians(a + azi_bias) for a in azimuth_list]
# pitch = elevation
pitchs = [math.radians(elevation_deg)] * len(azimuth_list)
extr, intr = _yaw_pitch_r_fov_to_extrinsics_intrinsics(yaws, pitchs, r, fov)
out = render_utils.render_frames(
sample,
extr,
intr,
{"resolution": resolution, "bg_color": bg_color, "backend": "gsplat"},
**kwargs,
)
return out # out["color"] shape = [N, H, W, 3]
And the inference workflow:
from utils.inference import Inference, render_video, make_scene
outputs = [self.inference(image_rgb, mask, seed=seed) for mask in masks]
# 5. Merge scene and prepare for rendering
scene_gs = make_scene(*outputs)
scene_gs_ready = ready_gaussian_for_video_rendering(scene_gs)
❓ Question
Is there a known mismatch between Blender's camera model and the _yaw_pitch_r_fov_to_extrinsics_intrinsics conversion used in SAM3D-Object?
Or is there an additional transformation needed to ensure the rendered views match Blender’s coordinate conventions?
Any guidance on how to correctly align the camera parameters would be greatly appreciated!
Contributor guide
Assessment
This issue has not been assessed yet.