Disallowed interactor causing offset issue

> **NOTE:** Any issue that does not follow the below template will be immediately closed and not re-opened until the template structure is adhered to.
### Environment
Unity 2021.3.16f

### Steps to reproduce
* Create two interactables, a cube and a sphere
* Set both to use follow grab, precision grab, no physics
* On the cube, assign the left interactor as a disallowed interactor
* Attempt to grab cube with left hand
* Then grab with right hand
* Observe rotation and position offset

[Video demo of issue](https://cdn.discordapp.com/attachments/717121338409484319/1250746304850559006/InteractorOffset.mp4?ex=666c0fc4&is=666abe44&hm=bdf1c10cba6a4db8fb311986aa59929ba308244d3cb0057217957fc433042f02&)

### Expected behavior
Right controller should not be impacted by disallowed interactor

### Current behavior
Interacting with left hand creates issue with right hand