Manipulating segmentation layer containing annotations

We have a need for manipulating structures (represented as a segmentation layer (annotation) link below). Somewhat similar to manipulating the transformation matrix (rotation, scaling + translation) under the source tab.

I was wondering if you had any suggestions on how we should go about implementing this and what sort of Interface method (i.e. key bindings/drag and drop) should be used.

Thanks,

Link to the viewer with our data: https://neuroglancer-demo.appspot.com/#!%7B%22dimensions%22:%7B%22x%22:%5B4.6000000000000004e-7%2C%22m%22%5D%2C%22y%22:%5B4.6000000000000004e-7%2C%22m%22%5D%2C%22z%22:%5B0.00002%2C%22m%22%5D%7D%2C%22position%22:%5B7874.16552734375%2C2444.890625%2C184.0110321044922%5D%2C%22crossSectionScale%22:9.018178748790614%2C%22projectionOrientation%22:%5B0.0019906109664589167%2C-0.0022145232651382685%2C0.007156445644795895%2C0.9999699592590332%5D%2C%22projectionScale%22:23605.130455376657%2C%22layers%22:%5B%7B%22type%22:%22image%22%2C%22source%22:%22precomputed://https://mousebrainatlas-datajoint-jp2k.s3.amazonaws.com/precomputed/MD589_fullres%22%2C%22name%22:%22image%22%7D%2C%7B%22type%22:%22segmentation%22%2C%22source%22:%22precomputed://https://test-bucket-sid.s3.amazonaws.com/final_precomputed_volumes/MD589%22%2C%22tab%22:%22source%22%2C%22name%22:%22annotation%22%7D%5D%2C%22selectedLayer%22:%7B%22layer%22:%22annotation%22%2C%22visible%22:true%2C%22size%22:599%7D%2C%22layout%22:%224panel%22%7D