ByteDance-Seed / ByteDance-Seed/Depth-Anything-3

Predicted Extrinsics Shape doesn't match the Extrinsics Input Shape

Open
#31 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.3k
Forks
702
PR merge metrics
No merged PRs in 30d

Description

Awesome work folks! Results are looking really good.

I am experimenting with some multi-view video footage and attempting to average the extrinsics predictions.
While doing so I realized that the predicted extrinsics matrix shape doesn't match that of the input for conditioning.

Extrinsics predicted shape - (2, 3, 4)
Extrinsics condition shape - (N, 4, 4)

It would be great to have both on the same standard.
Thanks!

## Example
This examples gives the following output:

```python
image_paths = [
"assets/examples/SOH/000.png",
"assets/examples/SOH/010.png"
]

prediction = model.inference(
image=image_paths,
process_res=504,
process_res_method="upper_bound_resize",
export_dir=None,
export_format="glb"
)
print(f"Depth shape: {prediction.depth.shape}")
print(f"Extrinsics: {prediction.extrinsics.shape if prediction.extrinsics is not None else 'None'}")
print(f"Intrinsics: {prediction.intrinsics.shape if prediction.intrinsics is not None else 'None'}")

prediction = model.inference(
image=image_paths,
extrinsics=prediction.extrinsics,
intrinsics=prediction.intrinsics,
process_res=504,
process_res_method="upper_bound_resize",
export_dir=None,
export_format="glb"
)
print(f"Final Depth shape: {prediction.depth.shape}")
print(f"Final Extrinsics: {prediction.extrinsics.shape if prediction.extrinsics is not None else 'None'}")
print(f"Final Intrinsics: {prediction.intrinsics.shape if prediction.intrinsics is not None else 'None'}")
```

## Output:

> [INFO ] Processed Images Done taking 0.03813338279724121 seconds. Shape: torch.Size([2, 3, 280, 504])
> [INFO ] Model Forward Pass Done. Time: 0.3240797519683838 seconds
> [INFO ] Conversion to Prediction Done. Time: 0.0009243488311767578 seconds
> Depth shape: (2, 280, 504)
> Extrinsics: (2, 3, 4)
> Intrinsics: (2, 3, 3)
> [INFO ] Processed Images Done taking 0.035338640213012695 seconds. Shape: torch.Size([2, 3, 280, 504])
>
> ---------------------------------------------------------------------------
> RuntimeError Traceback (most recent call last)
> Cell In[6], [line 21](vscode-notebook-cell:?execution_count=6&line=21)
> 17 print(f"Intrinsics: {prediction.intrinsics.shape if prediction.intrinsics is not None else 'None'}")
> 19 # Convert extrinsics
> ---> [21](vscode-notebook-cell:?execution_count=6&line=21) prediction = model.inference(
> 22 image=image_paths,
> 23 extrinsics=prediction.extrinsics,
> 24 intrinsics=prediction.intrinsics,
> 25 process_res=504,
> 26 process_res_method="upper_bound_resize",
> 27 export_dir=None,
> 28 export_format="glb"
> 29 )
> 30 print(f"Final Depth shape: {prediction.depth.shape}")
> 31 print(f"Final Extrinsics: {prediction.extrinsics.shape if prediction.extrinsics is not None else 'None'}")
>
> File ~/projects/lab/pose-detect/Depth-Anything-3/src/depth_anything_3/api.py:188, in DepthAnything3.inference(self, image, extrinsics, intrinsics, > align_to_input_ext_scale, infer_gs, render_exts, render_ixts, render_hw, process_res, process_res_method, export_dir, export_format, export_feat_layers, conf_thresh_percentile, num_max_points, show_cameras, feat_vis_fps, export_kwargs)
> 185 imgs, ex_t, in_t = self._prepare_model_inputs(imgs_cpu, extrinsics, intrinsics)
> 187 # Normalize extrinsics
> --> [188](https://vscode-remote+wsl-002bubuntu-005fjupyter.vscode-resource.vscode-cdn.net/home/gkann/projects/lab/pose-detect/Depth-Anything-3/notebooks/~/projects/lab/pose-detect/Depth-Anything-3/src/depth_anything_3/api.py:188) ex_t_norm = self._normalize_extrinsics(ex_t.clone() if ex_t is not None else None)
> 190 # Run model forward pass
> 191 export_feat_layers = list(export_feat_layers) if export_feat_layers is not None else []
>
> File ~/projects/lab/pose-detect/Depth-Anything-3/src/depth_anything_3/api.py:303, in DepthAnything3._normalize_extrinsics(self, ex_t)
> 301 return None
> 302 transform = affine_inverse(ex_t[:, :1])
> --> [303](https://vscode-remote+wsl-002bubuntu-005fjupyter.vscode-resource.vscode-cdn.net/home/gkann/projects/lab/pose-detect/Depth-Anything-3/notebooks/~/projects/lab/pose-detect/Depth-Anything-3/src/depth_anything_3/api.py:303) ex_t_norm = ex_t @ transform
> 304 c2ws = affine_inverse(ex_t_norm)
> 305 translations = c2ws[..., :3, 3]
>
> RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [2, 4] but got: [2, 3].

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.