Tensor API can't write .pcd files beyond 2.1 GB.
- Dominant language
- C++
- Stars
- 14k
- Forks
- 2.6k
- Avg merge
- 5d 18h
- Merged PRs (30d)
- 6
Description
### Checklist
- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).
### Describe the issue
The Tensor API fails to write .pcd point clouds with a nominal size of more than 2**31 *bytes*. It fails with a "MemoryError: std::bad_alloc" (the system is certainly not out of memory). The code snip below will reliably write a point cloud of N points and fail to write a point cloud of N+1 points.
The threshold of writable point clouds is about 2.1 GB. The example below demonstrates the behavior with 32-bit and 64-bit position-only point clouds at different point counts, but this byte size ceiling affects custom attributes exactly as you would predict (and is how I first encountered the issue).
A partial workaround for folks who *only* need position data is to convert to a legacy PointCloud and use the legacy writer. However, I require custom attributes and so I need to work with the new Tensor API.
This behavior was reproduced in Python 3.8 in Linux with both the Open3D release 0.17 and development 0.17.0+4d63591.
### Steps to reproduce the bug
```python
import numpy as np
import open3d as o3d
import open3d.core as o3c
def write_point_cloud_size_N(N, dtype):
""" Write a point cloud of N points, positions only. """
pc = o3d.t.geometry.PointCloud()
positions = np.random.random((N, 3)).astype(dtype)
pc.point.positions = o3c.Tensor.from_numpy(positions)
o3d.t.io.write_point_cloud('out.pcd', pc)
# 32-bit works at N pts. Three four-byte values per point.
N = 2**31 // (3 * 4)
write_point_cloud_size_N(N, np.float32)
# 32-bit FAILS at N+1 pts. Three four-byte values per point.
# Raises Exception "MemoryError: std::bad_alloc".
N = 2**31 // (3 * 4) + 1
write_point_cloud_size_N(N, np.float32)
# 64-bit works at M pts. Three eight-byte values per point.
N = 2**31 // (3 * 8)
write_point_cloud_size_N(N, np.float64)
# 64-bit FAILS at M+1 pts. Three eight-byte values per point.
N = 2**31 // (3 * 8) + 1
write_point_cloud_size_N(N, np.float64)
```
### Error message
MemoryError: std::bad_alloc
### Expected behavior
The Tensor API o3d.t.io.write_point_cloud writer should be able to write large .pcd files just as the legacy writer can.
### Open3D, Python and System information
```markdown
- Operating system: Ubuntu 20.04
- Python version: Python 3.8`
- Open3D version: release 0.17.0 and also development 0.17.0+4d63591
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: pip
```
### Additional information
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.