ByteDance-Seed / ByteDance-Seed/Depth-Anything-3
Inconsistent spatial resolution between `depth` and `ray` outputs prevents pointmap computation
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 702
- PR merge metrics
- No merged PRs in 30d
Description
## Problem Description
The `depth` and `ray` outputs from `DualDPT` have inconsistent spatial resolutions, which prevents direct computation of pointmaps (3D point clouds) using the unified depth-ray representation as described in the paper.
### Current Behavior
After forward pass in `DualDPT`, the outputs have mismatched shapes:
- `depth`: `[B, S, H, W]` = `[1, 20, 280, 504]` (3D tensor)
- `ray`: `[B, S, H, W, C]` = `[1, 20, 160, 288, 6]` (4D tensor)
**Key issues:**
1. **Spatial resolution mismatch**: `depth` has resolution 280×504, while `ray` has 160×288
2. **Cannot compute pointmap**: The spatial dimension mismatch prevents element-wise operations like `depth * ray_direction` to generate 3D points
### Expected Behavior
According to the paper's description of "unified depth-ray representation", `depth` and `ray` should have:
- **Same spatial resolution** (H, W) to enable direct pointmap computation
- Consistent tensor structure for element-wise operations
The pointmap should be computable as:
```python
# Expected: both have same spatial dimensions
points = t + depth[..., None] * ray[..., 3:] # ray direction (last 3 channels)
```
## Question
So how is L_P calculated in the loss function?
Additionally, when I use `origins = M[..., :3] # (B, N, H, W, 3)
t_c = origins.mean(dim=(-3, -2)) # (B, N, 3)`, the `t_c` I obtain differs significantly from the `t_c` estimated by `camera_head`. What could be the reason for this?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.