[Bug] SO-101 can make discontinuous joint jumps during Grip motion and stationary re-clutch

Describe the bug

When teleoperating an SO-101 with a Meta Quest 3S Touch Plus controller through CloudXR and Isaac Teleop, I observed sudden discontinuous robot motion in two situations:

1. A joint jump while continuously holding Grip/Squeeze and moving the controller.
2. A downward jump after moving the robot to a pose, releasing Grip, keeping the controller stationary, and pressing Grip again to re-engage the clutch.

The Quest controller pose and button input otherwise reached the Ubuntu host correctly. Grip/Squeeze acted as the clutch and Trigger controlled the gripper.

This was reproduced through the LeRobot SO-101 integration. I am filing it here because I would like guidance on the intended Isaac Teleop clutch/re-engagement behavior for underactuated arms and whether joint-continuity handling should be exposed or documented at the Isaac Teleop integration boundary.

To reproduce

1. Start CloudXR Runtime 6.2.0 on Ubuntu and connect a Quest 3S through the official CloudXR web client.
2. Connect an SO-101 follower and run the LeRobot Isaac Teleop example:

python -m examples.isaac_teleop_to_so101.teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/serial/by-id/<SO101_SERIAL_DEVICE> \
  --robot.id=so101_follower_main \
  --teleop.type=xr_controller \
  --reset_to_origin=false

3. Reproduction A:

   • Hold Grip continuously.
   • Move the right Quest controller through the workspace.
   • Observe that the arm can suddenly switch to a discontinuous elbow/wrist configuration.

4. Reproduction B:

   • Hold Grip and move the arm to a reachable pose.
   • Release Grip and confirm that the arm holds.
   • Keep the controller stationary.
   • Press Grip again.
   • Observe an immediate downward jump on re-engagement.

Expected behavior

• Motion should remain joint-continuous while Grip is held.
• Releasing Grip should hold the robot.
• Re-engaging Grip with a stationary controller should produce no robot movement on the engage frame.
• The following IK frames should continue from the measured/current joint configuration rather than switching to another local solution.

Observed behavior

In a quantified re-clutch run before the downstream fix, three physical Grip re-engagements produced first-frame maximum joint-target changes of:

• 6.422 degrees with 14.006 mm end-effector position change
• 3.116 degrees with 6.815 mm end-effector position change
• 3.186 degrees with 7.151 mm end-effector position change

The reported orientation change was 0 degrees in all three cases. The largest changes were mainly in elbow_flex, with wrist_flex also visibly involved in another event.

Videos

• Video 1 - jump while Grip is continuously held and the controller is moving:

  https://github.com/user-attachments/assets/f249e4d1-d16d-401f-91fb-02b7f3e34ddf

• Video 2 - downward jump when re-engaging Grip while the controller is stationary:

  https://github.com/user-attachments/assets/2e6d228d-d09b-4497-ae2c-a6e52b0d9927

Platforms

• Isaac Teleop Version: 1.3.131
• CloudXR Runtime: 6.2.0
• OS: Ubuntu 24.04 LTS, kernel 6.8.0-134-generic, x86_64
• GPU: NVIDIA GeForce RTX 4080, driver 595.71.05
• Integration: LeRobot v0.6.0, examples/isaac_teleop_to_so101
• XR Device: Meta Quest 3S with Touch Plus controllers
• Robot: SO-101 follower, 5-DoF arm plus gripper
• Control target rate: 30 Hz

Investigation and downstream workaround

The investigation in the LeRobot integration found two relevant behaviors:

• A calibrated/measured joint value can be slightly outside the URDF nominal limit (for example, shoulder_lift=-103 degrees while the URDF lower limit is approximately -100 degrees). A same-pose q -> FK(q) -> IK(FK(q)) operation can therefore project the state into the solver's feasible joint space instead of returning the measured joint configuration.
• On this 5-DoF arm, the same-pose projection may require multiple solves to settle, and a small first controller displacement near an underconstrained or near-singular pose can select a more distant local joint solution.

A downstream fix was implemented in commit f611bba, currently included in the independent draft LeRobot PR #4046.

The workaround keeps the measured arm joints exactly unchanged on the engage frame, pre-converges the same-pose IK state, adds native dq regularization, and rebases subsequent solver increments onto the measured joint configuration. The gripper remains an absolute 0-100 command.

After the fix, 47 physical re-engagements on the SO-101 produced a 0.000000-degree engage-frame joint jump, and the original elbow/wrist jump did not recur during small and larger XYZ motions.

Questions

1. Is there an intended Isaac Teleop pattern for guaranteeing joint-space continuity at clutch re-engagement on underactuated arms?
2. Should the clutch/integration API provide an engage-frame hold or joint-state rebase hook before Cartesian IK resumes?
3. Would NVIDIA prefer this behavior to remain entirely integration-specific, or should the Isaac Teleop documentation/example cover it?

Additional context

No hand tracking was used. The test used only the Quest 3S right Touch Plus controller. No cameras, data recording, or leader arm were involved.

Start with the examples.isaac_teleop_to_so101.teleoperate entry point and reproduce the two clutch scenarios using the command and platform details provided. Compare Isaac Teleop's behavior with downstream fix f611bba and determine whether an engage-frame hold or joint-state rebase belongs in the integration boundary; done means joint-continuous motion and no movement when re-engaging a stationary controller.