Polymetis server fails after two hours

## Type of Issue

Select the type of issue:
- [x] Bug report (to report a bug)
- [ ] Feature request (to request an additional feature)
- [ ] Tracker (I am just using this as a tracker)
- [ ] Refactor request
- [ ] Documentation Ask

## Description
Hi all,

I'm currently running polymetis with a Franka robot and the Robotiq gripper. Everything is working as intended, except for when we run polymetis for longer periods of time (~2 hrs). We'd like to be able to run it continuously for many hours or even days as we are testing reinforcement learning algorithms

The issue seems to occur within the server / robot client with the following command
`` launch_robot.py robot_client=franka_hardware robot_client.executable_cfg.robot_ip=172.16.0.2 ``
And everything will work as intended for two hours but eventually crash

## Current Behavior

When my training script crashes, I see the following output from the polymetis server
```
[2022-10-06 12:19:12.650] [error] Robot is unable to be controlled: libfranka: Move command aborted: motion aborted by reflex! ["cartesian_reflex"]
control_command_success_rate: 0.9
[2022-10-06 12:19:12.650] [warning] Performing automatic error recovery. This calls franka::Robot::automaticErrorRecovery, which is equivalent to pressing and releasing the external activation device.
[2022-10-06 12:19:12.650] [warning] Automatic error recovery attempt 1/3...
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
[2022-10-06 12:19:13.652] [warning] Robot operation recovered.
Get called, buffer size 73764767
Get called, buffer size 73764767
Get called, buffer size 73764767
[2022-10-06 12:19:13.655] [warning] Interrupted control update greater than threshold of 1000000000 ns. Reverting to default controller...
[2022-10-06 12:19:13.655] [info] Terminating custom controller, switching to default controller.
[2022-10-06 12:19:13.659] [warning] Tried to perform a controller update with no controller running.
```

## Expected Behavior

Not entirely sure what's to be expected, but when the experiment is running well, and the server is able to start the controller / switch to the default controller. For example, all I had to do with restart my training script and the server (with the same log as above) was able to continue working as follows:
```
[2022-10-06 12:19:13.659] [warning] Tried to perform a controller update with no controller running.
Get called, buffer size 77448624
[2022-10-06 13:20:47.203] [info] Loaded new controller.
Setting Torch policy to terminated.
[2022-10-06 13:20:49.181] [info] Terminating custom controller, switching to default controller.
Get called, buffer size 77450816
Get called, buffer size 77450821
Get called, buffer size 77450840
```

## Steps to reproduce

Not sure how to reproduce this other than maybe running a random policy for ~2hrs?