google-deepmind / google-deepmind/representations4d
Can not reproduce high-fidelity depth estimations
- Dominant language
- Jupyter Notebook
- Stars
- 238
- Forks
- 14
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 1
Description
Hi, thanks for publishing this awesome work @rishabhkabra!
I'm trying to follow this technical paradigm to develop a generalist video perception model. However, when I run the test script [genception_inference_demo.ipynb](https://github.com/google-deepmind/representations4d/blob/main/colabs/genception_inference_demo.ipynb) with the released genception_1.3b_transformers and genception_14b_transformers checkpoints, I've run into an issue.
The 14B model shows a clear improvement on depth predictions when viewed as video, but when I unproject the normalized depth into a point cloud, the result is very noisy:
The point cloud obtained from the 14B model is still noisy. Could you help me identify the problem?
Below is the denormalization code I use for depth:
`
def depth_normalize_inverse(
d_prime: np.ndarray,
alpha: float = _DEFAULT_DEPTH_ALPHA,
) -> np.ndarray:
"""Invert the nonlinear log-mapping to recover median-normalized depth.
Given d' = clip(α·log(d_norm + 1), 0, 1), the inverse is:
d_norm = exp(d' / α) - 1
The returned depth has **scale = 1.0** (median depth ≈ 1.0).
Args:
d_prime: Normalized depth in [0, 1], shape ``[H, W]`` or ``[T, H, W]``.
alpha: Must match the alpha used during forward normalization.
Returns:
Median-normalized depth (scale=1.0), same shape as input.
"""
d_prime = np.asarray(d_prime, dtype=np.float64)
d_norm = np.exp(d_prime / alpha) - 1.0
return d_norm.astype(np.float32)
def dit_depth_to_normalized(
dit_video: np.ndarray,
alpha: float = _DEFAULT_DEPTH_ALPHA,
) -> np.ndarray:
"""Convert model-output DIT depth video to normalized depth maps.
The GenCeption model outputs depth as a 3-channel uint8 video where the
three RGB channels are replicated copies of the same normalized value
(Section 3.3). This function reverses both the RGB encoding and the
nonlinear log-mapping to produce a single-channel depth map with
scale = 1.0 (median depth ≈ 1.0).
Args:
dit_video: Model output, uint8 ``[T, H, W, 3]`` or float
``[T, H, W, 3]`` in [-1, 1].
alpha: The log-mapping parameter used during training.
Returns:
Normalized depth, ``float32 [T, H, W]``, scale = 1.0.
"""
video = np.asarray(dit_video, dtype=np.float64)
# [-1, 1] → [0, 1]
video = (video + 1.0) / 2.0
video = video.clip(0.0, 1.0)
# Average across RGB channels (they are replicated for depth).
d_prime = video.mean(axis=-1) # [T, H, W]
# Invert the log-mapping.
depth_norm = depth_normalize_inverse(d_prime, alpha=alpha)
return depth_norm
`
Contributor guide
Research direction
Start with colabs/genception_inference_demo.ipynb and reproduce the reported outputs using the released genception_1.3b_transformers and genception_14b_transformers checkpoints. Inspect the supplied depth_normalize_inverse and dit_depth_to_normalized code alongside the notebook's depth and point-cloud steps; done means identifying why the 14B point cloud is noisy and documenting a reproducible correction or explanation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, numpy, python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100