Real-time mesh building

I am trying to use Open3D for real-time generation of 3D models using a Kinect for Azure device.
Now that there is CUDA support. I believe this should be possible.

To get a good start, I would appreciate any hints on how to get the most out of Open3D for this task.
I must add that I already know the camera pose of each image taken. With a small tolerance of about +/-30mm, and rotation +/-2deg.

I was already hinted to drop rgbd odometry and go directly to multiway registration. This will require input images converted to point clouds + image poses, and output an optimized pose graph with optimized image poses. This can then be used as input to a TSDF that can generate the final mesh. Is this the way to go?

The Kinect for Azure SDK can generate nice stable 30fps point clouds. Could I somehow take advantage of this?
The ScalableTSDFVolume Class only integrates rgbd images. Is there any way to fast register and integrate the point clouds, without needing the sensors RGBD-images for TSDF integration?