ByteDance-Seed / ByteDance-Seed/Depth-Anything-3

Inconsistent spatial resolution between `depth` and `ray` outputs prevents pointmap computation

Open
#101 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.3k
Forks
702
PR merge metrics
No merged PRs in 30d

Description

## Problem Description

The `depth` and `ray` outputs from `DualDPT` have inconsistent spatial resolutions, which prevents direct computation of pointmaps (3D point clouds) using the unified depth-ray representation as described in the paper.

### Current Behavior

After forward pass in `DualDPT`, the outputs have mismatched shapes:
- `depth`: `[B, S, H, W]` = `[1, 20, 280, 504]` (3D tensor)
- `ray`: `[B, S, H, W, C]` = `[1, 20, 160, 288, 6]` (4D tensor)

**Key issues:**
1. **Spatial resolution mismatch**: `depth` has resolution 280×504, while `ray` has 160×288
2. **Cannot compute pointmap**: The spatial dimension mismatch prevents element-wise operations like `depth * ray_direction` to generate 3D points

### Expected Behavior

According to the paper's description of "unified depth-ray representation", `depth` and `ray` should have:
- **Same spatial resolution** (H, W) to enable direct pointmap computation
- Consistent tensor structure for element-wise operations

The pointmap should be computable as:
```python
# Expected: both have same spatial dimensions
points = t + depth[..., None] * ray[..., 3:] # ray direction (last 3 channels)
```

## Question
Image
So how is L_P calculated in the loss function?

Additionally, when I use `origins = M[..., :3] # (B, N, H, W, 3)

t_c = origins.mean(dim=(-3, -2)) # (B, N, 3)`, the `t_c` I obtain differs significantly from the `t_c` estimated by `camera_head`. What could be the reason for this?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.