Geometric consistency across depth, ray, and camera outputs: clarification needed for metric applications

## Summary

First, congratulations on DA3 - the perceptual depth quality is excellent, and the unified depth-ray representation is an elegant concept. I'm investigating DA3 for metrological applications (3D reconstruction with known accuracy bounds) and have encountered several inconsistencies between outputs that prevent reliable metric use.

I've done a thorough code review to understand the implementation before raising these questions. I'd appreciate clarification on whether these are intended behaviors, documentation gaps, or areas where my understanding is incomplete.

---

## Core Issue: The outputs don't live in a consistent geometric reference frame

For metric applications, I need to be able to:

1. Unproject depth to 3D points using intrinsics
2. Use ray origins/directions to define the same 3D points
3. Have camera poses that are consistent with both

These three paths should converge to the same world-space geometry. Currently, they don't.

---

## Observations with Code Evidence

### 1. Ray origins are not constant per frame

For a pinhole camera, all rays originate from the camera center. The ray origin channels should be constant (or very nearly so) across all pixels in a frame—effectively encoding `t_c` redundantly at each pixel.

**Code review findings:**

The ray head outputs 7 channels with linear (unconstrained) activation:

```python
# src/depth_anything_3/model/dualdpt.py, lines 138-149
self.scratch.output_conv2_aux = nn.ModuleList([
nn.Sequential(
nn.Conv2d(head_features_1 // 2, head_features_2, kernel_size=3, stride=1, padding=1),
*ln_seq,
nn.ReLU(inplace=True),
nn.Conv2d(head_features_2, 7, kernel_size=1, stride=1, padding=0), # 7 channels: dir(3) + origin(3) + conf(1)
)
for _ in range(self.aux_levels)
])
```

The camera center is then computed as a weighted average of these spatially-varying origins:

```python
# src/depth_anything_3/utils/ray_utils.py, lines 495-500
T = torch.sum(camray[:, :, 3:] * confidence.unsqueeze(-1), dim=1) / torch.sum(
confidence, dim=-1, keepdim=True
)
```

I found no constraint in the codebase that enforces ray origins to be constant `[0,0,0]` in camera frame. In practice, `ray[:, :, 3:6].std(dim=(-2,-1))` shows significant spatial variation within frames.

**Question:** Is this intentional? Do ray origins encode something other than camera center, or should they be constrained?

---

### 2. Camera center estimates diverge between computation paths

Two independent paths compute camera center:

**Path A - Ray Head** (weighted average of ray origins):
```python
# src/depth_anything_3/utils/ray_utils.py, lines 495-500
T = torch.sum(camray[:, :, 3:] * confidence.unsqueeze(-1), dim=1) / torch.sum(
confidence, dim=-1, keepdim=True
)
```

**Path B - Camera Head** (direct network prediction):
```python
# src/depth_anything_3/model/cam_dec.py, lines 33-37
def forward(self, feat, camera_encoding=None, *args, **kwargs):
B, N = feat.shape[:2]
feat = feat.reshape(B * N, -1)
feat = self.backbone(feat)
out_t = self.fc_t(feat.float()).reshape(B, N, 3) # Camera center (translation)
```

These paths produce different values, and I found no constraint enforcing consistency between them.

**Note:** Path A computes `T` in camera coordinates (from ray origins), while Path B outputs world coordinates directly via `pose_encoding_to_extri_intri()` (`src/depth_anything_3/model/utils/transform.py`, lines 549-558). The coordinate frame difference may be intentional, but downstream code appears to use both without explicit reconciliation.

---

### 3. Scale factor convention: paper vs. code connection

The `apply_metric_scaling` function uses `scale_factor=300`:

```python
# src/depth_anything_3/utils/alignment.py, lines 118-133
def apply_metric_scaling(
depth: torch.Tensor, intrinsics: torch.Tensor, scale_factor: float = 300.0
) -> torch.Tensor:
focal_length = (intrinsics[:, :, 0, 0] + intrinsics[:, :, 1, 1]) / 2
return depth * (focal_length[:, :, None, None] / scale_factor)
```

Section 4.4 of the paper documents `f_c = 300` as the canonical focal length. The connection is clear once you know to look for it.

**However**, there's a naming collision. At inference, a *different* `scale_factor` is computed via least-squares alignment:

```python
# src/depth_anything_3/model/da3.py, lines 405-414
scale_factor = least_squares_scale_scalar(valid_metric_depth, valid_depth)
output.depth *= scale_factor
output.extrinsics[:, :, :3, 3] *= scale_factor
output.scale_factor = scale_factor.item() # This is NOT the 300 constant
```

Users encountering `prediction.scale_factor` may not realize this is a per-inference computed value, not the canonical `f_c = 300`. This likely explains the confusion in #94.

**Suggestion:** Rename one of these to avoid ambiguity (e.g., `canonical_focal_length` vs `alignment_scale_factor`).

---

### 4. Spatial resolution: decoder computes identical dimensions for both heads

Users report resolution mismatch between depth (280×504) and ray (160×288) outputs (#101). However, the decoder code computes identical output dimensions for both:

```python
# src/depth_anything_3/model/dualdpt.py, lines 233-234
h_out = int(ph * self.patch_size / self.down_ratio)
w_out = int(pw * self.patch_size / self.down_ratio)
```

Both heads use the same `down_ratio` parameter and the docstring confirms matching shapes:

```python
# src/depth_anything_3/model/dualdpt.py, lines 176-179
# Shapes:
# main: [B, S, out_dim, H/down_ratio, W/down_ratio]
# aux: [B, S, 7, H/down_ratio, W/down_ratio]
```

The mismatch users observe is **not explained by the released decoder code**. This suggests either:
- Runtime configuration not reflected in the repository
- Post-processing in training/inference pipeline
- Different model checkpoints with different configurations

**Question:** Under what configuration would depth and ray have different spatial dimensions?

---

### 5. Paper claims `d_cam = KR·d_I` but implementation uses standard `K⁻¹`

The paper (Section 3.1, near Equation 2) states: "The transformation from this canonical ray `d_I` to the ray direction `d_cam` in the target camera's coordinate system is given by `d_cam = KR·d_I`."

Standard pinhole geometry uses `d_cam = K⁻¹·p` for unprojection. The implementation follows the standard approach:

```python
# src/depth_anything_3/utils/geometry.py, lines 355-357
def inverse_intrinsic_matrix(ixts):
return torch.inverse(ixts)

# src/depth_anything_3/utils/geometry.py, lines 375-376
camera_space_points = torch.einsum(
"b v i j , h w j -> b v h w i", inverse_intrinsic_matrix(intrinsics), pixel_space_points
)
```

The homography estimation then maps from identity-K unprojected points to predicted rays:

```python
# src/depth_anything_3/utils/ray_utils.py, lines 459-493
I_cam_plane_unproj = unproject_depth(cam_plane_depth, I_K, ...) # Uses K⁻¹
R, focal_lengths, principal_points = compute_optimal_rotation_intrinsics_batch(
I_cam_plane_unproj, # src: identity K unprojected points (via K⁻¹)
camray[:, :, :3], # dst: predicted ray directions
...
)
```

The implementation is correct (standard pinhole geometry). The paper's description may be a notational shorthand that doesn't literally describe the code path. This is minor, but caused initial confusion when trying to understand the architecture.

---

## The Deeper Question

DA3 optimizes multiple objectives that should be geometrically redundant:

```
L = L_D(D̂,D) + L_M(R̂,M) + L_P(D̂⊙d+t,P) + βL_C(ĉ,v) + αL_grad(D̂,D)
```

At inference, each head produces independent predictions. The network has learned to minimize these losses *on average across the training distribution*, not to maintain *per-sample geometric consistency*.

This is the difference between:
- **Statistical consistency**: outputs are correct in expectation over the data distribution
- **Geometric consistency**: outputs satisfy projective geometry constraints for each individual sample

For metrological applications, I need the latter. Is there a recommended approach for enforcing geometric consistency at inference time, or is this fundamentally outside the design goals of DA3?

---

## Questions for Maintainers

1. **Ray origins**: Are they intended to be constant per frame (camera center), or do they encode per-pixel information intentionally?

2. **Camera center paths**: Should the ray-derived and camera-head-derived centers agree? If so, is there a recommended post-processing step?

3. **Scale factor naming**: Would you consider renaming to disambiguate the canonical `f_c=300` from the inference-time alignment factor?

4. **Resolution mismatch**: What configuration produces the different resolutions users report in #101?

5. **COLMAP export**: Given the ray/camera-head divergence, would using external SfM poses with DA3 depth produce more reliable reconstructions than using `export_to_colmap`?

---

## Suggested Improvements

If geometric consistency is a goal:

1. **Clarify scale factor naming**: Rename to distinguish `canonical_focal_length=300` from `alignment_scale_factor`

2. **Add docstrings**: Link `apply_metric_scaling` to Section 4.4's `f_c` explanation

3. **Consistency check**: Add optional validation comparing ray-derived vs camera-head camera centers

4. **Constrain ray origins**: Consider adding a loss term or post-processing to enforce constant ray origins per frame (if that's the intended behavior)

---

## Environment

- **DA3 version**: latest main branch (commit reviewed: current HEAD)
- **Model**: DA3-Nested-Giant-Large and DA3-Metric-Large
- **Use case**: Metrological 3D reconstruction with accuracy requirements
- **Code files reviewed**:
- `src/depth_anything_3/model/dualdpt.py`
- `src/depth_anything_3/model/da3.py`
- `src/depth_anything_3/model/cam_dec.py`
- `src/depth_anything_3/utils/ray_utils.py`
- `src/depth_anything_3/utils/geometry.py`
- `src/depth_anything_3/utils/alignment.py`
- `src/depth_anything_3/utils/export/colmap.py`

---

Thank you for any clarification. I'm happy to run specific experiments, provide diagnostic outputs, or test proposed fixes.