ByteDance-Seed / ByteDance-Seed/Depth-Anything-3
Combining DA3-Metric Depth with DA3-Nested Pose for Multi-view Reconstruction?
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 702
- PR merge metrics
- No merged PRs in 30d
Description
Hi, and thanks for the great work on DA3, I'm new to the reconstruction field and this is truly impressive.
I’m trying to reconstruct a small indoor scene from 3 views with low overlap. Visual correctness matters more than strict metric accuracy at this stage.
**My setup is:** Single moving camera + Known intrinsics, unknown extrinsics + 3 views, low overlap (for max information).
I tried to compare reconstruction results on DA3-METRIC-LARGE and DA3NESTED-GIANT-LARGE.
**My observation:** DA3-METRIC-LARGE produces very crisp depth maps, often better along edges, especially on objects farther away from the camera, compared to DA3NESTED-GIANT-LARGE-1.1.
I am converting to metric depth using `metric_depth = net_output * fx / 300`.
**For 3D point cloud reconstruction I use:**
Metric:
```
depth_all = prediction_metric.depth # from metric model
K_all = intrinsics # from ground truth, scaled to our image
Extrinsics_all = prediction.extrinsics # from nested model
images_all = prediction_metric.processed_images # Already numpy uint8 array
```
Nested:
```
depth_all = prediction.depth
K_all = prediction.intrinsics
Extrinsics_all = prediction.extrinsics
images_all = prediction.processed_images
```
**Result:**
- Per-view point clouds from the metric model look much better and accurate to the real-world depths
- Multi-view alignment is more consistent with the nested model, whereas merging multi-views of the metric's point clouds results in much less accurate alignment (poses are coarsely correct, but do not fit together: this makes sense as I'm using the nested models pose estimation, and that pose estimation fits with the nested predictions since the model is trained for relative depth / pose. )
- Trying to use the 3 views from metric and run ICP / registration fails due to low overlap
Is there a recommended way to combine: metric model depth quality with the nested model pose consistency to get the best of both worlds?
Or do I need to have a fixed rig with 3 calibrated cameras and use both known intrinsic and extrinsic information to acquire the desired result?
Is this a known limitation given low overlap and missing pose estimation in the metric model?
Any pointers would be greatly appreciated.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.