ByteDance-Seed / ByteDance-Seed/Depth-Anything-3

[BUG] Gaussians' means is not properly calculated.

Open
#247 4 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.3k
Forks
702
PR merge metrics
No merged PRs in 30d

Description

### Bug Report: Inconsistent Depth Representation in `GaussianAdapter` Leading to Distorted Gaussians

#### Summary

Issues #236 and #100 report distortion in Gaussian representations produced by `GaussianAdapter`. After investigation, the root cause appears to be inconsistent handling of depth representations (z-depth vs. ray distance), along with a potentially misbehaving `depth_offset`.

---

#### Observations

1. **Mismatch in Depth Representation**

* `GaussianAdapter.forward` takes `output.depth` as input.
* This depth is **z-depth** (camera coordinate), but later computations treat it as **ray distance** when deriving Gaussian means.
* This inconsistency leads to incorrect spatial positioning.

2. **Incorrect Use of `depth_offset`**

* `depth_offset = raw_gaussians[..., -1]` is directly added to `gs_depths`.
* Evidence suggests `depth_offset` is likely defined in **ray distance space**, causing further inconsistency when combined with z-depth.

3. **Empirical Validation**
Several fixes were tested:

* Converting `depth_offset` into z-depth offset before adding → still incorrect
* Converting `output.depth` to ray distance before adding `depth_offset` → still incorrect

These results suggest:

* The issue is not only representation mismatch
* `depth_offset` itself is likely **poorly learned or unstable**, contributing significantly to the distortion

---

#### Proposed Fix

1. **Use z-depth for unprojection**

* Compute Gaussian means via **z-depth-based unprojection**, ensuring consistency with the original depth definition.

2. **Disable `depth_offset`**

* Given its unstable behavior and lack of reliable contribution, removing or disabling `depth_offset` improves correctness.

---

#### Additional Notes

* The problem likely stems from mixed assumptions about depth parameterization during training vs. inference.
* A more robust solution may require revisiting how `depth_offset` is supervised or parameterized.

---

#### Expected Outcome

Applying the above fixes should eliminate the observed Gaussian distortion and produce geometrically consistent Gaussian means.

---

Please let me know if further experiments or ablations are needed.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at GaussianAdapter.forward and trace how output.depth, gs_depths, raw_gaussians[..., -1], and depth_offset are combined when deriving Gaussian means. Compare the z-depth unprojection path with the proposed removal or disabling of depth_offset; done means the resulting Gaussian means are geometrically consistent without the observed distortion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.