Requesting a "Stopping" instance to stop results in eventual running state

While stress testing create / destroy VMs on the colo rack (R11) I encountered the case where my tooling emits a instance_stop request to instances independent of their state. If this request is sent to a VM that is already in a stopping state then it fails with:

Stopping instance 8ebe8c46-a8f2-483d-a8c7-fdee07d7b86b failed due to Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "d0067857-5fc9-4418-aa93-8aa5369484ea", "content-length": "166", "date": "Thu, 10 Oct 2024 16:48:15 GMT"}; value: Error { error_code: None, message: "The instance was running but is no longer reachable. It is being moved to the Failed state.", request_id: "d0067857-5fc9-4418-aa93-8aa5369484ea" }

It makes sense that the call failed, but maybe not as a 503? It seems more like a 4xx error.

The primary issue though is that this leads to the VM being moved to the failed state. Auto-restart makes this more confusing for the user, as once the VM hits the failed state it is then recovered resulting in a running VM.

It seems instead that this request should fail with a 400 error and not perform any action.

Start at the instance_stop request entry point and trace how a request for an instance already in the stopping state is handled. Reproduce the transition described for VM 8ebe8c46-a8f2-483a-8c7c-fdee07d7b86b, then verify that the request returns a client error without moving the instance to failed or allowing auto-restart to return it to running.