facebookresearch / facebookresearch/sam-3d-objects

slat_decoder_mesh failed.

Open
#166 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.4k
Forks
878
PR merge metrics
No merged PRs in 30d

Description

## Question description
I'm able to access and visualize outputs["gs"] correctly via the [supersplat editor](https://superspl.at/editor), but output["glb"] returns empty. Tracing this back to the decode_slat function:

device: NVIDIA GeForce RTX 4090 (48GB)

```python
def decode_slat(
self,
slat: sp.SparseTensor,
formats: List[str] = ["mesh", "gaussian"],
) -> dict:
"""
Decode the structured latent.

Args:
slat (sp.SparseTensor): The structured latent.
formats (List[str]): The formats to decode the structured latent to.

Returns:
dict: The decoded structured latent.
"""
logger.info("Decoding sparse latent...")
ret = {}
with torch.no_grad():
if "mesh" in formats:
ret["mesh"] = self.models["slat_decoder_mesh"](slat)
if "gaussian" in formats:
ret["gaussian"] = self.models["slat_decoder_gs"](slat)
if "gaussian_4" in formats:
ret["gaussian_4"] = self.models["slat_decoder_gs_4"](slat)
# if "radiance_field" in formats:
# ret["radiance_field"] = self.models["slat_decoder_rf"](slat)
return ret
```
the issue occurs at ret["mesh"] = self.models["slat_decoder_mesh"](slat). Going further, I found that NaN values appear during the forward pass of
'sam3d_objects.model.backbone.tdfy_dit.models.structured_latent_vae.decoder_mesh.SLatMeshDecoderTdfyWrapper'. I have checked the model weight loading, and the weights are loaded correctly." Has anyone encountered similar issues? Any suggestions on what might cause NaN only in the mesh decoder?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.