ByteDance-Seed / ByteDance-Seed/Depth-Anything-3

[Bug] Gaussian point clouds are not scaled during metric depth alignment, causing pose inconsistency

Open Beginner friendly
#242 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.3k
Forks
702
PR merge metrics
No merged PRs in 30d

Description

## Problem Description

In `NestedDepthAnything3Net._apply_depth_alignment()`, when converting predicted depth to metric scale, the code scales `depth` and `extrinsics` but **misses scaling the `gaussians`**. This causes the exported Gaussian point clouds (via `gs_ply`) to have a different scale than the camera poses (via `cameras.json`), resulting in incorrect rendering when using standard 3DGS viewers.

## Root Cause

In `src/depth_anything_3/model/da3.py`, lines 410-414:

```python
# Apply scaling to depth and extrinsics
output.depth *= scale_factor
output.extrinsics[:, :, :3, 3] *= scale_factor
output.is_metric = 1
output.scale_factor = scale_factor.item()
```

The `output.gaussians` (which contains `means` and `scales`) is not scaled, even though it was computed using the pre-scaling depth values.

## Impact

When exporting to 3DGS format (`--export-format gs_ply`):
- **Before fix**: Gaussian points have ~40x larger extent than camera positions
- **Result**: Rendering from predicted camera poses shows completely wrong views (PSNR ~7 dB)
- **Verification**: 3DGS viewers cannot correctly display the scene from the exported cameras

## Proposed Fix

Add scaling for gaussians in `_apply_depth_alignment()`:

```python
# Apply scaling to depth and extrinsics
output.depth *= scale_factor
output.extrinsics[:, :, :3, 3] *= scale_factor
output.is_metric = 1
output.scale_factor = scale_factor.item()

# FIX: Also scale gaussians if they exist to maintain consistency
if hasattr(output, 'gaussians') and output.gaussians is not None:
output.gaussians.means *= scale_factor
output.gaussians.scales *= scale_factor
```

## Files Affected

- `src/depth_anything_3/model/da3.py`: `_apply_depth_alignment()` method (around line 416)

## Additional Context

This issue only affects users who:
1. Use the nested/metric model (DA3NESTED variants)
2. Export to 3DGS format (`gs_ply`)
3. Attempt to render from the exported camera poses

The `scene.glb` export is not affected because it uses COLMAP point clouds which are computed from the scaled depth maps.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read src/depth_anything_3/model/da3.py at NestedDepthAnything3Net._apply_depth_alignment(), starting with the metric scaling block around line 416. Check how gaussians stores means and scales, then verify a DA3NESTED export with --export-format gs_ply keeps Gaussian points and cameras.json in the same scale. Done means the exported scene renders correctly from the predicted camera poses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.