Azure Kinect create_from_rgbd_image PCD scale difference when working with depth intrinsics

I've been experimenting with working with downscaling the color image to depth instead of the default in all the examples which is the other way around (`k4a_transformation_color_image_to_depth_camera` instead of `k4a_transformation_depth_image_to_color_camera`) to see if I can get any performance gains if working with less data.

The problem is after calling `create_from_rgbd_image` with the depth cameras intrinsics and extrinsics I get a pointcloud that is ~8-9% smaller in scale than compared to working with the default color camera's image size / intrinsics. I probably missed a step somewhere as these should line up perfectly in theory?

What I do:
* open a mkv recording
* take raw depth frame as it was recorded
* take color image transformed with `k4a_transformation_color_image_to_depth_camera`
* retrieve intrinsics (only for depth camera) as described here: https://github.com/intel-isl/Open3D/blob/9e7617cea4406b64df966097d501a26245fe7559/cpp/open3d/io/sensor/azure_kinect/MKVReader.cpp#L96-L107
* retreive depth width & height from a lookup table for depth mode, as described in the spec: https://docs.microsoft.com/en-us/azure/kinect-dk/hardware-specification#depth-camera-supported-operating-modes
* retreive extrinsics from calibration data (pretty sure it's correct, as the angles line up, this shouldn't affect the scale anyway - only rotation + translation?)
* call `pcd.create_from_rgbd_image` with rgbd, intrinsics and extrinsics
* dump to ply

Not the best capture but should illustrate the problem. Generated from the same frame of data, light is in color camera's space, dark in depth's:

Original color space intrinsic.json:

{
"color_mode" : "MJPG_1536P",
"depth_mode" : "NFOV_UNBINNED",
"height" : 1536,
"intrinsic_matrix" :
[
962.31732177734375,
0.0,
0.0,
0.0,
961.93304443359375,
0.0,
1023.294189453125,
777.87109375,
1.0
],
"serial_number_" : "000002594912",
"stream_length_usec" : 27984323,
"width" : 2048
}

Modified depth space intrinsic.json:

{
"color_mode" : "MJPG_1536P",
"depth_mode" : "NFOV_UNBINNED",
"height" : 576,
"intrinsic_matrix" :
[
504.14996337890625,
0.0,
0.0,
0.0,
504.07647705078125,
0.0,
319.98883056640625,
344.919921875,
1.0
],
"serial_number_" : "000002594912",
"stream_length_usec" : 27984323,
"width" : 640
}