ByteDance-Seed / ByteDance-Seed/Depth-Anything-3
[Bug] Gaussian point clouds are not scaled during metric depth alignment, causing pose inconsistency
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 702
- PR merge metrics
- No merged PRs in 30d
Description
## Problem Description
In `NestedDepthAnything3Net._apply_depth_alignment()`, when converting predicted depth to metric scale, the code scales `depth` and `extrinsics` but **misses scaling the `gaussians`**. This causes the exported Gaussian point clouds (via `gs_ply`) to have a different scale than the camera poses (via `cameras.json`), resulting in incorrect rendering when using standard 3DGS viewers.
## Root Cause
In `src/depth_anything_3/model/da3.py`, lines 410-414:
```python
# Apply scaling to depth and extrinsics
output.depth *= scale_factor
output.extrinsics[:, :, :3, 3] *= scale_factor
output.is_metric = 1
output.scale_factor = scale_factor.item()
```
The `output.gaussians` (which contains `means` and `scales`) is not scaled, even though it was computed using the pre-scaling depth values.
## Impact
When exporting to 3DGS format (`--export-format gs_ply`):
- **Before fix**: Gaussian points have ~40x larger extent than camera positions
- **Result**: Rendering from predicted camera poses shows completely wrong views (PSNR ~7 dB)
- **Verification**: 3DGS viewers cannot correctly display the scene from the exported cameras
## Proposed Fix
Add scaling for gaussians in `_apply_depth_alignment()`:
```python
# Apply scaling to depth and extrinsics
output.depth *= scale_factor
output.extrinsics[:, :, :3, 3] *= scale_factor
output.is_metric = 1
output.scale_factor = scale_factor.item()
# FIX: Also scale gaussians if they exist to maintain consistency
if hasattr(output, 'gaussians') and output.gaussians is not None:
output.gaussians.means *= scale_factor
output.gaussians.scales *= scale_factor
```
## Files Affected
- `src/depth_anything_3/model/da3.py`: `_apply_depth_alignment()` method (around line 416)
## Additional Context
This issue only affects users who:
1. Use the nested/metric model (DA3NESTED variants)
2. Export to 3DGS format (`gs_ply`)
3. Attempt to render from the exported camera poses
The `scene.glb` export is not affected because it uses COLMAP point clouds which are computed from the scaled depth maps.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read src/depth_anything_3/model/da3.py at NestedDepthAnything3Net._apply_depth_alignment(), starting with the metric scaling block around line 416. Check how gaussians stores means and scales, then verify a DA3NESTED export with --export-format gs_ply keeps Gaussian points and cameras.json in the same scale. Done means the exported scene renders correctly from the predicted camera poses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100