# About metric-scale depth estimation and point cloud reconstruction

## Problems
I have tried to use both DA3NESTED-GIANT-LARGE, DA3NESTED-GIANT-LARGE-1.1, DA3METRIC-LARGE on my data. However, the results aren't good as expected (I run inference to get npz file -> run my script below to get ply file).

For instance, below are my reconstructed point clouds using the model DA3NESTED-GIANT-LARGE, DA3NESTED-GIANT-LARGE-1.1, DA3METRIC-LARGE, note that the true scale is 9cm x 16.2cm. However, the result that I get was ~ 0.166 x 0.278 (In theory, if the depth is correctly metric-scale, and given camera's intrinsic, in pixels, I can back-project 2D depth image into metric-scale 3D point cloud). I will put my implementation below.

If my implementation is correct, I guess this is probably the limitation of neural-based model on metric-scale depth estimation, which is failing when being applied to out-of-domain data. And perhaps, fine-tuning is the only way to mitigate this.

I really hope that I made the mistake somewhere, anyone met a problem similar this? can I get your recommendation on how to get correct metric-scale 3D point clouds.

## RECONSTRUCTED POINT CLOUD
### DA3NESTED-GIANT-LARGE

Note that the outputs by DA3NESTED-GIANT-LARGE and LARGE1.1 have already been in meters.

### DA3NESTED-GIANT-LARGE-1.1

### DA3METRIC-LARGE

This model is even worse compared to two aforementioned models, since it predicted the wrong scale {width x height}.

## MY IMPLEMENTATION TO TRANSFER RESULT.NPZ TO PLY FILE

```
import numpy as np
import cv2
import os

# ─────────────────────────────────────────────
# 1. LOAD
# ─────────────────────────────────────────────
# data = np.load()
depth = data['depth'].squeeze(0) # (H_depth, W_depth)

# rgb = cv2.imread()
rgb = cv2.cvtColor(rgb, cv2.COLOR_BGR2RGB)

# ─────────────────────────────────────────────
# 2. INTRINSICS — original image resolution
# ─────────────────────────────────────────────
H_orig, W_orig = 1080, 1920
fx = 1360.637451
fy = 1359.588745
cx = 967.345520
cy = 537.803223

# ─────────────────────────────────────────────
# 3. SCALE INTRINSICS TO DEPTH RESOLUTION
# ─────────────────────────────────────────────
H_depth, W_depth = depth.shape
print(f"Depth resolution : {H_depth} x {W_depth}")
print(f"RGB resolution : {rgb.shape[0]} x {rgb.shape[1]}")

fx_s = fx * (W_depth / W_orig)
fy_s = fy * (H_depth / H_orig)
cx_s = cx * (W_depth / W_orig)
cy_s = cy * (H_depth / H_orig)
print(f"Scaled intrinsics: fx={fx_s:.3f}, fy={fy_s:.3f}, cx={cx_s:.3f}, cy={cy_s:.3f}")

# ─────────────────────────────────────────────
# 4. RESIZE RGB TO MATCH DEPTH RESOLUTION
# ─────────────────────────────────────────────
rgb_resized = cv2.resize(rgb, (W_depth, H_depth), interpolation=cv2.INTER_LINEAR)

# ─────────────────────────────────────────────
# 5. BUILD PIXEL GRID
# ─────────────────────────────────────────────
u = np.arange(W_depth)
v = np.arange(H_depth)
uu, vv = np.meshgrid(u, v) # both (H_depth, W_depth)

# ─────────────────────────────────────────────
# 6. UNPROJECT TO 3D
# ─────────────────────────────────────────────
Z = depth
X = (uu - cx_s) * Z / fx_s
Y = (vv - cy_s) * Z / fy_s

# ─────────────────────────────────────────────
# 7. FILTER INVALID DEPTHS
# ─────────────────────────────────────────────
valid = (Z > 0) & np.isfinite(Z)
X = X[valid]
Y = Y[valid]
Z = Z[valid]
colors = rgb_resized[valid].astype(np.uint8)
points = np.stack([X, Y, Z], axis=-1) # (N, 3)
print(f"Total valid points: {len(points)}")
print(f"Depth range: {Z.min():.3f} ~ {Z.max():.3f} m")

# ─────────────────────────────────────────────
# 8. SAVE AS PLY WITH COLOR
# ─────────────────────────────────────────────
def save_ply_rgb(filepath, points, colors):
os.makedirs(os.path.dirname(filepath), exist_ok=True)
header = (
"ply\n"
"format ascii 1.0\n"
f"element vertex {len(points)}\n"
"property float x\n"
"property float y\n"
"property float z\n"
"property uchar red\n"
"property uchar green\n"
"property uchar blue\n"
"end_header\n"
)
with open(filepath, 'w') as f:
f.write(header)
for p, c in zip(points, colors):
f.write(f"{p[0]:.6f} {p[1]:.6f} {p[2]:.6f} {c[0]} {c[1]} {c[2]}\n")
print(f"Saved → {filepath}")

save_ply_rgb(PATH, points, colors)

```