oxidecomputer / oxidecomputer/omicron
500 error when trying to attach an allocated floating IP address under race conditions
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This is arguably a corner case. When I have two simultaneous instance requests (made using Terraform) that try to use the same floating IP, the second request fails with a 500 error:
│ Error: Error creating instance
│
│ with oxide_instance.app[1],
│ on app.tf line 68, in resource "oxide_instance" "app":
│ 68: resource "oxide_instance" "app" {
│
│ API error: POST https://silo11.sys.rack2.eng.oxide.computer/v1/instances?project=be6cc6d9-8b58-437f-ad95-c27d7c38bbcf
│ ----------- RESPONSE -----------
│ Status: 500 Internal
│ Message: Internal Server Error
│ RequestID: 84733981-d8b8-48ce-9701-ac89bafab388
│ ------- RESPONSE HEADERS -------
│ Content-Type: [application/json]
│ X-Request-Id: [84733981-d8b8-48ce-9701-ac89bafab388]
│ Date: [Fri, 12 Jan 2024 23:39:47 GMT]
│ Content-Length: [124]
These are the nexus log lines for 500 error:
21:27:07.907Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (dropshot_external): request completed
error_message_external = Internal Server Error
error_message_internal = saga ACTION error at node "output": Object (of type ByName("mailserver")) not found: floating-ip
file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841
latency_us = 1208944
local_addr = 172.30.2.5:443
method = POST
remote_addr = 172.20.17.42:65326
req_id = f27a9983-8dce-400a-a57e-4db273599c86
response_code = 500
uri = https://silo11.sys.rack2.eng.oxide.computer/v1/instances?project=be6cc6d9-8b58-437f-ad95-c27d7c38bbcf
The nature of the error should be HTTP 400 (user error). It'll be nice to be able to distinguish the invalid floating ip name error from internal system errors. For comparison, this is the error I get when attempting to attach the same disk to two concurrent VM requests:
│ Error: Error creating instance
│
│ with oxide_instance.app[1],
│ on app.tf line 68, in resource "oxide_instance" "app":
│ 68: resource "oxide_instance" "app" {
│
│ API error: POST https://silo11.sys.rack2.eng.oxide.computer/v1/instances?project=be6cc6d9-8b58-437f-ad95-c27d7c38bbcf
│ ----------- RESPONSE -----------
│ Status: 400 InvalidRequest
│ Message: cannot attach disk "app-0": disk is attached to another instance
│ RequestID: a8a873ab-6ceb-4168-a59a-4adaebc9e672
│ ------- RESPONSE HEADERS -------
│ Content-Type: [application/json]
│ X-Request-Id: [a8a873ab-6ceb-4168-a59a-4adaebc9e672]
│ Date: [Fri, 12 Jan 2024 23:43:32 GMT]
│ Content-Length: [175]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the concurrent instance requests shown in app.tf line 68 and trace the Nexus handling for the POST /v1/instances endpoint. Compare the floating-IP race path with the concurrent disk-attachment error; done means the invalid floating-IP conflict returns HTTP 400 with a user-facing message instead of HTTP 500.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100