isl-org / isl-org/Open3D

Point Cloud projection to RGBD introduces artifacts in color image when CUDA is used

Open
#6,612 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](https://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](https://www.open3d.org/docs/release/) and the [latest documentation](https://www.open3d.org/docs/latest/) (for `main` branch).

### Describe the issue

I am using `o3d.t.geometry.PointCloud.project_to_rgbd_image()` and have found strange and problematic black-pixel artifacts in the color result _only_ when using CUDA - they do not appear on CPU. The depth image appears normal at the corresponding pixels.

I first observed this running on a live feed of a motionless scene -- the artifacts were fairly constant but did change somewhat with the changing sensor noise. Below I'm including code for reproducing the effect on a fake point cloud.

The point cloud is generated on an xy grid and the projection comes from an overhead (z-direction) camera. I offset the point z coordinates either randomly or as a function of radius.
- Random --> artifact locations also appear random
- Radial --> artifacts curiously occupy one diagonal half of the view.

![o3d17_rand](https://github.com/isl-org/Open3D/assets/129541401/9b289072-1d87-40d0-807e-858a10e6702e)
![o3d18dev](https://github.com/isl-org/Open3D/assets/129541401/697b4cef-43d5-48ca-8750-1b7839a80134)

### Steps to reproduce the bug

```python
import matplotlib.pyplot as plt
import numpy as np
import open3d as o3d

# Parameters
size_m = 0.25 # Size of square image in meters
size_px = 64 # Size of square image in pixels
size_pts = 300 # Number of points in created pointcloud (gets squared)
camera_height_m = 10
device_names = ["CPU:0", "CUDA:0"]
height_mode = "random" # in ["random", "radial"]

# Derived values
half_size_px = size_px // 2
m_per_px = size_m / size_px
focal_length_px = camera_height_m / m_per_px
projection_kwargs = {
'width': size_px,
'height': size_px,
'extrinsics': o3d.core.Tensor([ # 10m above origin, pointing straight down
[1, 0, 0, 0],
[0, -1, 0, 0],
[0, 0, -1, camera_height_m],
[0, 0, 0, 1],
]),
'intrinsics': o3d.core.Tensor([ # Such that contents of size_m exactly fill the image
[focal_length_px, 0, half_size_px],
[0, focal_length_px, half_size_px],
[0, 0, 1],
]),
'depth_scale': 1.0,
'depth_max': ((focal_length_px * m_per_px) / np.tan(half_size_px / focal_length_px)),
}

# Making the point cloud
m_per_px = size_m / size_pts
coords_xy_m = m_per_px * (np.transpose(np.mgrid[0:size_pts, 0:size_pts], axes=[1, 2, 0]) - (size_pts // 2))

# Add radial variation in height
radius_m = np.sqrt(np.sum(coords_xy_m**2, axis=-1))
normalized_weight = radius_m / radius_m.max()
if height_mode == "radial":
coords_z_m = 0.1*(1 - normalized_weight) # 10cm high in the middle, 0 at corners
else:
coords_z_m = np.random.randn(size_pts, size_pts)**2*0.1
coords_m = np.concatenate([coords_xy_m, coords_z_m[..., np.newaxis]], axis=-1)

# Add variation in color from bottom-left to top-right
pcd = o3d.t.geometry.PointCloud(o3d.core.Tensor(coords_m.reshape((-1, 3)), dtype=o3d.core.float32))
color_weights = coords_xy_m.sum(axis=-1)
color_weights = (color_weights - color_weights.min()) / (color_weights.max() - color_weights.min())
colors = np.dstack([color_weights, color_weights*0, 1-color_weights]).reshape((-1, 3))
print(colors.min(), colors.max())
pcd.point.colors = o3d.core.Tensor(colors, dtype=o3d.core.float32)

fig, axs = plt.subplots(2, len(device_names), figsize=(5, 5))
axs[0,0].set_ylabel("depth")
axs[1,0].set_ylabel("color")
for axs_col, device_name in zip(axs.T, device_names):
axs_col[0].set_title(device_name)
pcd = pcd.to(device=o3d.core.Device(device_name))
image = pcd.project_to_rgbd_image(**projection_kwargs)
axs_col[0].imshow(image.depth.cpu())
axs_col[1].imshow(image.color.cpu())

plt.show()
```

### Error message

None.

### Expected behavior

I expect that, since all points in the point cloud have a valid color value, any pixel with a depth value should also have a color value. I also cannot infer any reasonable way that the radial height distribution should relate to the diagonal split distribution of the artifacts.

### Open3D, Python and System information

```markdown
- Operating system: Ubuntu 22.04
- Python version: Python 3.10.12
- Open3D version: 0.18.0+0cf605f, 0.18, 0.17
- System architecture: x86
- Is this a remote workstation?: yes
- How did you install Open3D?: pip
- Compiler version: n/a
```

### Additional information

Machine 1 GPU:
```
NVIDIA RTX A5000
Driver Version: 530.30.02
CUDA Version: 12.1
```

Machine 2 GPU:
```
NVIDIA RTX A6000
Driver Version: 545.23.08
CUDA Version: 12.3
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.