facebookresearch / facebookresearch/sam-3d-objects

Problem of demo_3db_mesh_alignment: Object GLB from sam_3d_objects has large scale/translation mismatch vs. 3DB human mesh

Open
#98 2 comments 4 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.4k
Forks
878
PR merge metrics
No merged PRs in 30d

Description

Hi! I’m trying to reproduce the `demo_3db_mesh_alignment` pipeline on a new single image. I can generate valid human outputs from **sam-3d-body** (mask / mesh / focal length), and I can also generate an object mesh (`.glb`) from **sam_3d_objects** using a mask.

However, when I run the final alignment/visualization step, the human pose looks reasonable, but the object position and size are **significantly off** (large translation + scale mismatch). The object ends up far away / too large or too small compared to the human (see screenshot).
[test_data.zip](https://github.com/user-attachments/files/23963198/test_data.zip)
Image
Image

---

## Reproduction Steps

### 1) Generate 3DB results (human) from `sam-3d-body`

From my single image, I generated:

- `3DB_results/mask_human.png`
- `3DB_results/human.ply`
- `3DB_results/focal_length.json`

### 2) Generate object mesh from `sam_3d_objects`

I used a bowl mask to infer a mesh and exported it as GLB:

```python
import sys
sys.path.append("notebook")
from inference import Inference, load_image, load_mask

cfg = "checkpoints/hf/pipeline.yaml"
infer = Inference(cfg, compile=False)

image = load_image("/root/code/DexGrasp/data/test_data/bowl.png") # (H,W,3 or 4)
mask = load_mask("/root/code/DexGrasp/data/test_data/mask_bowl.png") # bool array

output = infer(image, mask, seed=0)

glb = output["glb"] # trimesh.Trimesh
glb.export("/root/code/DexGrasp/data/test_data/bowl.glb")
```
### 3) Run demo_3db_mesh_alignment-style alignment + visualization
I then ran my alignment code:
```python
success, output_mesh_path, result = process_and_save_alignment(
mesh_path="/test_data/meshes/human_object/3DB_results/human.ply",
mask_path="/test_data/meshes/human_object/3DB_results/mask_human.png",
image_path="/test_data/bowl.png",
output_dir="/test_data/meshes/human_object/aligned_meshes",
device=device,
focal_length_json_path="/test_data/meshes/human_object/3DB_results/focal_length.json"
)

demo, combined_glb_path = visualize_meshes_interactive(
aligned_mesh_path="/test_data/meshes/human_object/3DB_results/human.ply",
dfy_mesh_path="/test_data/bowl.glb",
share=True

)

```
##

Questions
1. Is there any required normalization for sam_3d_objects outputs (e.g., unit scale, coordinate axis, camera center) before combining with 3DB human results?

2. Could this be caused by focal length / intrinsics mismatch between sam-3d-body’s focal_length.json and sam_3d_objects’ reconstruction?

3. Is there a recommended way to export/convert the object GLB so its scale/origin matches the alignment pipeline?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.