oxidecomputer / oxidecomputer/omicron

500 error when trying to attach an allocated floating IP address under race conditions

Open
#4,811 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This is arguably a corner case. When I have two simultaneous instance requests (made using Terraform) that try to use the same floating IP, the second request fails with a 500 error:

│ Error: Error creating instance
│ 
│   with oxide_instance.app[1],
│   on app.tf line 68, in resource "oxide_instance" "app":
│   68: resource "oxide_instance" "app" {
│ 
│ API error: POST https://silo11.sys.rack2.eng.oxide.computer/v1/instances?project=be6cc6d9-8b58-437f-ad95-c27d7c38bbcf
│ ----------- RESPONSE -----------
│ Status: 500 Internal
│ Message: Internal Server Error
│ RequestID: 84733981-d8b8-48ce-9701-ac89bafab388
│ ------- RESPONSE HEADERS -------
│ Content-Type: [application/json]
│ X-Request-Id: [84733981-d8b8-48ce-9701-ac89bafab388]
│ Date: [Fri, 12 Jan 2024 23:39:47 GMT]
│ Content-Length: [124]

These are the nexus log lines for 500 error:

21:27:07.907Z INFO 65a11c18-7f59-41ac-b9e7-680627f996e7 (dropshot_external): request completed
    error_message_external = Internal Server Error
    error_message_internal = saga ACTION error at node "output": Object (of type ByName("mailserver")) not found: floating-ip
    file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/ff87a01/dropshot/src/server.rs:841
    latency_us = 1208944
    local_addr = 172.30.2.5:443
    method = POST
    remote_addr = 172.20.17.42:65326
    req_id = f27a9983-8dce-400a-a57e-4db273599c86
    response_code = 500
    uri = https://silo11.sys.rack2.eng.oxide.computer/v1/instances?project=be6cc6d9-8b58-437f-ad95-c27d7c38bbcf

The nature of the error should be HTTP 400 (user error). It'll be nice to be able to distinguish the invalid floating ip name error from internal system errors. For comparison, this is the error I get when attempting to attach the same disk to two concurrent VM requests:

│ Error: Error creating instance
│ 
│   with oxide_instance.app[1],
│   on app.tf line 68, in resource "oxide_instance" "app":
│   68: resource "oxide_instance" "app" {
│ 
│ API error: POST https://silo11.sys.rack2.eng.oxide.computer/v1/instances?project=be6cc6d9-8b58-437f-ad95-c27d7c38bbcf
│ ----------- RESPONSE -----------
│ Status: 400 InvalidRequest
│ Message: cannot attach disk "app-0": disk is attached to another instance
│ RequestID: a8a873ab-6ceb-4168-a59a-4adaebc9e672
│ ------- RESPONSE HEADERS -------
│ Content-Type: [application/json]
│ X-Request-Id: [a8a873ab-6ceb-4168-a59a-4adaebc9e672]
│ Date: [Fri, 12 Jan 2024 23:43:32 GMT]
│ Content-Length: [175]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the concurrent instance requests shown in app.tf line 68 and trace the Nexus handling for the POST /v1/instances endpoint. Compare the floating-IP race path with the concurrent disk-attachment error; done means the invalid floating-IP conflict returns HTTP 400 with a user-facing message instead of HTTP 500.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.