isl-org / isl-org/Open3D

Tensor API can't write .pcd files beyond 2.1 GB.

Open
#6,043 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
14k
Forks
2.6k
Avg merge
5d 18h
Merged PRs (30d)
6

Description

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### Describe the issue

The Tensor API fails to write .pcd point clouds with a nominal size of more than 2**31 *bytes*. It fails with a "MemoryError: std::bad_alloc" (the system is certainly not out of memory). The code snip below will reliably write a point cloud of N points and fail to write a point cloud of N+1 points.

The threshold of writable point clouds is about 2.1 GB. The example below demonstrates the behavior with 32-bit and 64-bit position-only point clouds at different point counts, but this byte size ceiling affects custom attributes exactly as you would predict (and is how I first encountered the issue).

A partial workaround for folks who *only* need position data is to convert to a legacy PointCloud and use the legacy writer. However, I require custom attributes and so I need to work with the new Tensor API.

This behavior was reproduced in Python 3.8 in Linux with both the Open3D release 0.17 and development 0.17.0+4d63591.

### Steps to reproduce the bug

```python
import numpy as np
import open3d as o3d
import open3d.core as o3c

def write_point_cloud_size_N(N, dtype):
""" Write a point cloud of N points, positions only. """
pc = o3d.t.geometry.PointCloud()
positions = np.random.random((N, 3)).astype(dtype)
pc.point.positions = o3c.Tensor.from_numpy(positions)
o3d.t.io.write_point_cloud('out.pcd', pc)

# 32-bit works at N pts. Three four-byte values per point.
N = 2**31 // (3 * 4)
write_point_cloud_size_N(N, np.float32)

# 32-bit FAILS at N+1 pts. Three four-byte values per point.
# Raises Exception "MemoryError: std::bad_alloc".
N = 2**31 // (3 * 4) + 1
write_point_cloud_size_N(N, np.float32)

# 64-bit works at M pts. Three eight-byte values per point.
N = 2**31 // (3 * 8)
write_point_cloud_size_N(N, np.float64)

# 64-bit FAILS at M+1 pts. Three eight-byte values per point.
N = 2**31 // (3 * 8) + 1
write_point_cloud_size_N(N, np.float64)
```

### Error message

MemoryError: std::bad_alloc

### Expected behavior

The Tensor API o3d.t.io.write_point_cloud writer should be able to write large .pcd files just as the legacy writer can.

### Open3D, Python and System information

```markdown
- Operating system: Ubuntu 20.04
- Python version: Python 3.8`
- Open3D version: release 0.17.0 and also development 0.17.0+4d63591
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: pip
```

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.