`RemoveFromScene` action causes Unity to freeze when modifying scenes

Hello, thank you so much for your amazing work on AI2-THOR!

I'm currently trying to customize the existing scenes from AI2-THOR, by removing some objects and adding my desired objects on the receptacles. To do so, I first teleported the agent to the point where it face the shelvingunit receptacle, and tried to remove the bottle from the receptacle.

However, I'm running into an issue when using the `RemoveFromScene` action. I've tested this both with `start_unity=True` and `start_unity=False`, and in both cases, the simulation seems to freeze after the action is called.

In particular, when `start_unity=True`, the object disappears from the Unity window (as expected), but the simulation hangs and does not proceed beyond that point.

Here's a simplified version of the Python code I’m using:
```python
OPEN_UNITY = True
# OPEN_UNITY = False

if OPEN_UNITY:
unity_open_kwargs = {'start_unity': True}
else:
unity_open_kwargs = {
'start_unity': False,
'port': 8200,
'server_class': ai2thor.wsgi_server.WsgiServer,
}

controller = Controller(
commit_id=THOR_COMMIT_ID,
**unity_open_kwargs,
scene=SCENE_DICT['scene_name'],
...
)

event = controller.step(
action='Teleport',
position=SCENE_DICT['agent_state']['position'],
...
)

# Get receptacles and placed objects
visible_objects = get_visible_objects_from_mask(event)
visible_receptacles = [obj for obj in visible_objects if obj['receptacle']]
placedObjectsIds = [o for ro in visible_receptacles for o in ro['receptacleObjectIds']]

# Try removing an object
target_obj_id = 'Bottle|+01.54|+00.89|-02.54'
event = controller.step('RemoveFromScene', objectId=target_obj_id)
```
This is the log from the python script:
```
Traceback (most recent call last):
File "/lib/python3.10/site-packages/ai2thor/controller.py", line 1070, in step
self.last_event = self.server.receive()
File "/lib/python3.10/site-packages/ai2thor/fifo_server.py", line 262, in receive
metadata, files = self._recv_message(
File "lib/python3.10/site-packages/ai2thor/fifo_server.py", line 179, in _recv_message
header = self._read_with_timeout(
File "lib/python3.10/site-packages/ai2thor/fifo_server.py", line 147, in _read_with_timeout
raise TimeoutError(
TimeoutError: Reading from AI2-THOR backend timed out (using 100.0s) timeout.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "custom_object.py", line 122, in
event = controller.step('RemoveFromScene', objectId='Bottle|+01.54|+00.89|-02.54')
File "lib/python3.10/site-packages/ai2thor/controller.py", line 1092, in step
raise (TimeoutError if isinstance(e, TimeoutError) else RuntimeError)(
TimeoutError: Error encountered when running action {'action': 'RemoveFromScene', 'objectId': 'Bottle|+01.54|+00.89|-02.54', 'sequenceId': 3} in scene FloorPlan1_physics.
```
Before I call "RemoveFromScene":


After:

Do you have any insights into what might be causing this issue? Could it be related to removing objects that are currently visible or interactable?

Thank you again for your support and for providing such an amazing tool!