ByteDance-Seed / ByteDance-Seed/Depth-Anything-3
[BUG] Gaussians' means is not properly calculated.
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 702
- PR merge metrics
- No merged PRs in 30d
Description
### Bug Report: Inconsistent Depth Representation in `GaussianAdapter` Leading to Distorted Gaussians
#### Summary
Issues #236 and #100 report distortion in Gaussian representations produced by `GaussianAdapter`. After investigation, the root cause appears to be inconsistent handling of depth representations (z-depth vs. ray distance), along with a potentially misbehaving `depth_offset`.
---
#### Observations
1. **Mismatch in Depth Representation**
* `GaussianAdapter.forward` takes `output.depth` as input.
* This depth is **z-depth** (camera coordinate), but later computations treat it as **ray distance** when deriving Gaussian means.
* This inconsistency leads to incorrect spatial positioning.
2. **Incorrect Use of `depth_offset`**
* `depth_offset = raw_gaussians[..., -1]` is directly added to `gs_depths`.
* Evidence suggests `depth_offset` is likely defined in **ray distance space**, causing further inconsistency when combined with z-depth.
3. **Empirical Validation**
Several fixes were tested:
* Converting `depth_offset` into z-depth offset before adding → still incorrect
* Converting `output.depth` to ray distance before adding `depth_offset` → still incorrect
These results suggest:
* The issue is not only representation mismatch
* `depth_offset` itself is likely **poorly learned or unstable**, contributing significantly to the distortion
---
#### Proposed Fix
1. **Use z-depth for unprojection**
* Compute Gaussian means via **z-depth-based unprojection**, ensuring consistency with the original depth definition.
2. **Disable `depth_offset`**
* Given its unstable behavior and lack of reliable contribution, removing or disabling `depth_offset` improves correctness.
---
#### Additional Notes
* The problem likely stems from mixed assumptions about depth parameterization during training vs. inference.
* A more robust solution may require revisiting how `depth_offset` is supervised or parameterized.
---
#### Expected Outcome
Applying the above fixes should eliminate the observed Gaussian distortion and produce geometrically consistent Gaussian means.
---
Please let me know if further experiments or ablations are needed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at GaussianAdapter.forward and trace how output.depth, gs_depths, raw_gaussians[..., -1], and depth_offset are combined when deriving Gaussian means. Compare the z-depth unprojection path with the proposed removal or disabling of depth_offset; done means the resulting Gaussian means are geometrically consistent without the observed distortion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100