google-deepmind / google-deepmind/representations4d

Can not reproduce high-fidelity depth estimations

Open
#9 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
238
Forks
14
Avg merge
5d 18h
Merged PRs (30d)
1

Description

Hi, thanks for publishing this awesome work @rishabhkabra!

I'm trying to follow this technical paradigm to develop a generalist video perception model. However, when I run the test script [genception_inference_demo.ipynb](https://github.com/google-deepmind/representations4d/blob/main/colabs/genception_inference_demo.ipynb) with the released genception_1.3b_transformers and genception_14b_transformers checkpoints, I've run into an issue.

The 14B model shows a clear improvement on depth predictions when viewed as video, but when I unproject the normalized depth into a point cloud, the result is very noisy:

Image

The point cloud obtained from the 14B model is still noisy. Could you help me identify the problem?

Image

Below is the denormalization code I use for depth:

`
def depth_normalize_inverse(
d_prime: np.ndarray,
alpha: float = _DEFAULT_DEPTH_ALPHA,
) -> np.ndarray:
"""Invert the nonlinear log-mapping to recover median-normalized depth.

Given d' = clip(α·log(d_norm + 1), 0, 1), the inverse is:
d_norm = exp(d' / α) - 1

The returned depth has **scale = 1.0** (median depth ≈ 1.0).

Args:
d_prime: Normalized depth in [0, 1], shape ``[H, W]`` or ``[T, H, W]``.
alpha: Must match the alpha used during forward normalization.

Returns:
Median-normalized depth (scale=1.0), same shape as input.
"""
d_prime = np.asarray(d_prime, dtype=np.float64)
d_norm = np.exp(d_prime / alpha) - 1.0
return d_norm.astype(np.float32)

def dit_depth_to_normalized(
dit_video: np.ndarray,
alpha: float = _DEFAULT_DEPTH_ALPHA,
) -> np.ndarray:
"""Convert model-output DIT depth video to normalized depth maps.

The GenCeption model outputs depth as a 3-channel uint8 video where the
three RGB channels are replicated copies of the same normalized value
(Section 3.3). This function reverses both the RGB encoding and the
nonlinear log-mapping to produce a single-channel depth map with
scale = 1.0 (median depth ≈ 1.0).

Args:
dit_video: Model output, uint8 ``[T, H, W, 3]`` or float
``[T, H, W, 3]`` in [-1, 1].
alpha: The log-mapping parameter used during training.

Returns:
Normalized depth, ``float32 [T, H, W]``, scale = 1.0.
"""
video = np.asarray(dit_video, dtype=np.float64)

# [-1, 1] → [0, 1]
video = (video + 1.0) / 2.0
video = video.clip(0.0, 1.0)

# Average across RGB channels (they are replicated for depth).
d_prime = video.mean(axis=-1) # [T, H, W]

# Invert the log-mapping.
depth_norm = depth_normalize_inverse(d_prime, alpha=alpha)
return depth_norm
`

Contributor guide

Open the contributing guide

Research direction

Start with colabs/genception_inference_demo.ipynb and reproduce the reported outputs using the released genception_1.3b_transformers and genception_14b_transformers checkpoints. Inspect the supplied depth_normalize_inverse and dit_depth_to_normalized code alongside the notebook's depth and point-cloud steps; done means identifying why the 14B point cloud is noisy and documenting a reproducible correction or explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, numpy, python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.