Misaligned pointcloud from a singleway view.

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### My Question

I am learning Open3D from zero and I am trying to do a 3D interior scene reconstruction from a public dataset - [GMU Kitchens](https://cs.gmu.edu/~robot/gmu-kitchens.html) - and I get this misaligned result that looks like it is working almost properly but has like 4-5 different relatively aligned sets of points that are not aligned with all the other sets.

I have used default config and then played with it for a while but I could do get much change.
It is probably a newbie mistake and a fix but I don't seem to be able to catch it.