isl-org / isl-org/Open3D

Segmemtation Fault when calling hidden_point_removal from pytorch DataLoader with num_workers >0

Open
#7,233 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [x] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [x] For Python issues, I have tested with the [latest development wheel](https://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [x] I have checked the [release documentation](https://www.open3d.org/docs/release/) and the [latest documentation](https://www.open3d.org/docs/latest/) (for `main` branch).

### Describe the issue

Calling `hidden_point_removal` from my pytorch DataLoader worker process causes an unexpected segmentation fault when `num_workers` >0. When using `num_workers` = 0, there is no problem. I call it from the `getitem` method of my dataset.
I found a similar issue (#6425 ) and tried to set the multiprocessing context to both `'forkserver'` and `'spawn'`, but it did not help. The error still happens at the same point of running. (around the same epochs)

### Steps to reproduce the bug

```python
def simulate_occlusion(points, param):
"""Occlusion simulation using Open3D's hidden point removal."""
import open3d as o3d

if points.shape[0] == 0:
raise ValueError("Input points array is empty.")

try:
original_points = points.copy()

scale = 10**param # Cap at 10¹ instead of 10²

# Camera positioning with limits
base_radius = 3
camera_dist = base_radius + np.random.uniform(-0.5, 0.5)

phi = np.arccos(1 - 2 * np.random.rand())
theta = np.random.uniform(0, 2 * np.pi)

camera_location = np.array([
camera_dist * np.sin(phi) * np.cos(theta),
camera_dist * np.sin(phi) * np.sin(theta),
camera_dist * np.cos(phi)
])

# Create point cloud
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(points[:, :3])

# Calculate HPR radius with safety limits
max_dist = np.max(np.linalg.norm(points[:, :3] - camera_location, axis=1))
hpr_radius = max_dist * scale

# Safety check for radius
if hpr_radius <= 0 or np.isinf(hpr_radius) or np.isnan(hpr_radius):
raise ValueError("Invalid HPR radius calculated.")

# Perform HPR with timeout protection
_, pt_map = pcd.hidden_point_removal(camera_location, hpr_radius)

if len(pt_map) == 0:
return original_points

return points[pt_map, :]

except Exception as e:
# On any error, return the original points
print(f"Warning: Error in simulate_occlusion: {e}, returning original points")
return points

# I create a dataloader like this:
import torch.multiprocessing as mp
mp.set_start_method('forkserver', force=True)
...
train_loader = torch.utils.data.DataLoader(
train_dataset, batch_sampler=train_batch_sampler,
num_workers=args.workers, pin_memory=True,
collate_fn=customized_collate_fn)

# the simulate_occlusion method is called like this:
def _load_and_preprocess_objaverse_points(self, file_path):
"""Loads points from the Objaverse NPZ file"""
try:
with np.load(file_path) as data:
point_set = data['point_cloud']
except Exception as e:
raise ValueError(f"Error loading Objaverse point cloud file {file_path}: {str(e)}")

# Normalize the point cloud coordinates
point_set[:, 0:3] = pc_normalize(point_set[:, 0:3])

# simulate occlusion
if self.simulate_occlusion:
point_set = simulate_occlusion(point_set, param=2)

# Randomly subsample to get nuscenes_median points
if point_set.shape[0] > self.nuscenes_median:
indices = np.random.choice(point_set.shape[0], self.nuscenes_median, replace=False)
point_set = point_set[indices, :]

# Visualize the point cloud using Open3D (debugging)
#o3d.visualization.draw_geometries([o3d.geometry.PointCloud(o3d.utility.Vector3dVector(point_set[:, :3]))])

# Add height dimension if needed
if self.use_height:
gravity_dim = 1 # Assuming Y-axis is up
height_array = point_set[:, gravity_dim:gravity_dim + 1] - point_set[:, gravity_dim].min()
point_set = np.concatenate((point_set, height_array), axis=1)

return point_set
```

### Error message

ERROR: Unexpected segmentation fault encountered in worker.
[rank2]: Traceback (most recent call last):
[rank2]: File "/home/x_gerna/.conda/envs/ulip/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1131, in _try_get_data
[rank2]: data = self._data_queue.get(timeout=timeout)
[rank2]: File "/home/x_gerna/.conda/envs/ulip/lib/python3.10/queue.py", line 179, in get
[rank2]: raise Empty
[rank2]: _queue.Empty

[rank2]: The above exception was the direct cause of the following exception:

[rank2]: Traceback (most recent call last):
[rank2]: File "/proj/berzelius-2023-364/users/x_gerna/repos/ULIP-Outdoors/main.py", line 712, in
[rank2]: main(args)
[rank2]: File "/proj/berzelius-2023-364/users/x_gerna/repos/ULIP-Outdoors/main.py", line 310, in main
[rank2]: train_stats = train(train_loader, model, criterion, optimizer, scaler, epoch, lr_schedule, lr_schedule_low, args)
[rank2]: File "/proj/berzelius-2023-364/users/x_gerna/repos/ULIP-Outdoors/main.py", line 383, in train
[rank2]: for data_iter, inputs in enumerate(train_loader):
[rank2]: File "/home/x_gerna/.conda/envs/ulip/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
[rank2]: data = self._next_data()
[rank2]: File "/home/x_gerna/.conda/envs/ulip/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1327, in _next_data
[rank2]: idx, data = self._get_data()
[rank2]: File "/home/x_gerna/.conda/envs/ulip/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1283, in _get_data
[rank2]: success, data = self._try_get_data()
[rank2]: File "/home/x_gerna/.conda/envs/ulip/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1144, in _try_get_data
[rank2]: raise RuntimeError(f'DataLoader worker (pid(s) {pids_str}) exited unexpectedly') from e
[rank2]: RuntimeError: DataLoader worker (pid(s) 1595307) exited unexpectedly

### Expected behavior

Run without segmentation faults with `num_workers` >0 in the DataLoader

### Open3D, Python and System information

```markdown
- Operating system: Red Hat Enterprise Linux 8.8
- Python version: 3.10.17 | packaged by conda-forge | (main, Apr 10 2025, 22:19:12) [GCC 13.3.0]
- Open3D version: output from python: 0.19.0+0969dfb
- System architecture: x86
- Is this a remote workstation?: yes
- How did you install Open3D?: conda
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.