Front in headless renderer/customized visualization not working the same as Eye in offscreen renderer.

### Checklist

- [X] I have searched for [similar issues](https://github.com/isl-org/Open3D/issues).
- [X] For Python issues, I have tested with the [latest development wheel](http://www.open3d.org/docs/latest/getting_started.html#development-version-pip).
- [X] I have checked the [release documentation](http://www.open3d.org/docs/release/) and the [latest documentation](http://www.open3d.org/docs/latest/) (for `master` branch).

### My Question

I have finally gotten the headless rendering up and running on my Jetson Xavier, but I am confused about how to use the set_front() function, compared to the "eye" variable in the Offscreen renderer.
I was expecting it to act the same, so if my lookat z and front z are the same then I would get a side profile of the object.
When I run my code as shown below (It was a b#@¤h to format) the two first images are correct, where I am looking straight from the top and bottom, but the next four images(front, back, left and right) are still from the top, but slightly shifted to their respective sides.
```
import open3d as o3d
from matplotlib import pyplot as plt
def headless_renderer(obj):
headless_renderer.vis = o3d.visualization.Visualizer()
headless_renderer.index = -1
# Lookat front up
trajectories = [[[0, 0, 3.5394509849096005], [0, 0, 4.392000000069565], [0, 1, 0]],
[[0, 0, 3.5394509849096005], [0, 0, 2.686901969749636], [0, 1, 0]],
[[0, 0, 3.5394509849096005], [0, -0.7385355480931748, 3.5394509849096005], [0, 0, 1]],
[[0, 0, 3.5394509849096005], [0, 0.744177384321476, 3.5394509849096005], [0, 0, -1]],
[[0, 0, 3.5394509849096005], [-0.7615882973737071, 0, 3.5394509849096005], [0, 0, 1]],
[[0, 0, 3.5394509849096005], [0.7530830284707681, 0, 3.5394509849096005], [0, 0, 1]]]

def move_forward(vis):
ctr = vis.get_view_control()
glb = headless_renderer
if glb.index >= 0:
depth = vis.capture_screen_float_buffer()
plt.imshow(depth); plt.title(""); plt.show()
glb.index += 1
if glb.index < len(trajectories):
ctr.set_lookat(trajectories[glb.index][0])
print(f"setting front to: {trajectories[glb.index][1]}")
ctr.set_front(trajectories[glb.index][1])
ctr.set_up(trajectories[glb.index][2])
ctr.set_zoom(0.6999)
else:
headless_renderer.vis.destroy_window()
vis = headless_renderer.vis
vis.create_window("renderer", 256, 256)
vis.add_geometry(obj)
vis.get_render_option().load_from_json("/home/open3d_data/extract/DemoCustomVisualization/renderoption.json")
vis.register_animation_callback(move_forward)
vis.run()
cvx_hull = o3d.geometry.TriangleMesh.create_coordinate_frame(size=0.1, origin=[0.0, 0.0, obj_center])
headless_renderer(cvx_hull)
```
How do I use the set_front() compared to the eye variable in Offscreenrenderer.setup_camera()?