Program get stuck when I use ray_cast

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](https://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](https://www.open3d.org/docs/release/) and the [latest documentation](https://www.open3d.org/docs/latest/) (for `main` branch).

### My Question

I'm trying to implement TSDF integration. When I followed the example code, everything works fine. However, when I used my own data, I found the program would stuck at [`ray_cast`](https://www.open3d.org/docs/release/python_api/open3d.t.geometry.VoxelBlockGrid.html#open3d.t.geometry.VoxelBlockGrid.ray_cast) or [`synthesize_model_frame`](https://www.open3d.org/docs/release/python_api/open3d.t.pipelines.slam.Model.html#open3d.t.pipelines.slam.Model.synthesize_model_frame).
It's strange that sometimes raycast would complete quite fast, sometimes the program would stuck. The input images are the same.
I have no idea what went wrong... :(