oxidecomputer / oxidecomputer/omicron

External networking API allows addition of conflicting address lots and IP addresses

Open
#11,010 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

networking
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

In testing #10961, I made some networking changes to dublin that Nexus allowed but I think should not have. I initially set dublin up using BGP with qsfp0 configured on both switches:

[rack_network_config]
infra_ip_first = "172.20.15.61"
infra_ip_last = "172.20.15.62"

[[rack_network_config.bgp]]
asn = 65106
originate = ["172.20.29.0/24","172.20.36.0/24"]

[rack_network_config.switch0.qsfp0]
routes = []
addresses = [{address = "172.20.15.61/29"}]
uplink_port_speed = "100G"
uplink_port_fec = "rs"
autoneg = false
bgp_peers = [{asn = 65106, addr = "172.20.15.59", port = "qsfp0"}]

[rack_network_config.switch1.qsfp0]
routes = []
addresses = [{address = "172.20.15.62/29"}]
uplink_port_speed = "100G"
uplink_port_fec = "rs"
autoneg = false
bgp_peers = [{asn = 65106, addr = "172.20.15.59", port = "qsfp0"}]

I wanted to try adding a static address / route to switch1/qsfp1, and stumbled forward blindly using some configs that accidentally overlapped with the above, but Nexus allowed me to set up things that it seems like it should not have. When I initially tried to add the address, I got an address unavailable, which is perfectly reasonable:

% oxide system networking addr add --rack $rack --switch switch1 --port qsfp0 --addr 172.20.15.62/29 --lot initial-infra
Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "c5fc8ff4-7f18-4c0f-a4a4-06bcedd0c731", "content-length": "128", "date": "Wed, 05 Aug 2026 18:00:40 GMT"}; value: Error { error_code: Some("InvalidRequest"), message: "address unavailable", request_id: "c5fc8ff4-7f18-4c0f-a4a4-06bcedd0c731" }

However, I was able to create a different address lot with the same addresses:

% cat add-static-routing.json
{
    "name": "john-static",
    "description": "address lot for static routing",
    "kind": "infra",
    "blocks": [
        {
            "first_address": "172.20.15.61",
            "last_address": "172.20.15.62"
        }
    ]
}
% oxide system networking address-lot create --name john-static --description 'adding static routing' --json-body add-static-routing.json
{
  "blocks": [
    {
      "first_address": "172.20.15.61",
      "id": "b202fea9-a857-4055-92b6-d460e3a6b43e",
      "last_address": "172.20.15.62"
    }
  ],
  "lot": {
    "description": "address lot for static routing",
    "id": "eb537eab-0e08-4fa8-8eca-263165cd1ec7",
    "kind": "infra",
    "name": "john-static",
    "time_created": "2026-08-05T18:01:58.478720Z",
    "time_modified": "2026-08-05T18:01:58.478720Z"
  }
}

This already seems bogus, because we have two different address lots covering the same IPs in CRDB now:

root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from address_lot;
                   id                  |     name      |               description                |         time_created          |         time_modified         | time_deleted | kind
---------------------------------------+---------------+------------------------------------------+-------------------------------+-------------------------------+--------------+--------
  eb537eab-0e08-4fa8-8eca-263165cd1ec7 | john-static   | address lot for static routing           | 2026-08-05 18:01:58.47872+00  | 2026-08-05 18:01:58.47872+00  | NULL         | infra
  f14e49a5-4299-445e-9853-17614778b2d9 | initial-infra | initial infrastructure ip address lot    | 2026-08-04 21:30:45.603884+00 | 2026-08-04 21:30:45.603884+00 | NULL         | infra
  f5041974-fb81-48b7-8124-25dbee726713 | as65106-lot   | Address lot for announce set in as 65106 | 2026-08-04 21:30:45.662304+00 | 2026-08-04 21:30:45.662304+00 | NULL         | infra
(3 rows)

root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from address_lot_block;
                   id                  |            address_lot_id            | first_address | last_address
---------------------------------------+--------------------------------------+---------------+----------------
  1b2b84c0-1c31-4538-bf3e-bc605fe655b6 | f5041974-fb81-48b7-8124-25dbee726713 | 172.20.36.0   | 172.20.36.255
  56625a05-357c-41ef-a4b4-a90e81432167 | f5041974-fb81-48b7-8124-25dbee726713 | 172.20.29.0   | 172.20.29.255
  b202fea9-a857-4055-92b6-d460e3a6b43e | eb537eab-0e08-4fa8-8eca-263165cd1ec7 | 172.20.15.61  | 172.20.15.62
  df4c0a42-afd6-4392-a79e-2a0951f85c09 | f14e49a5-4299-445e-9853-17614778b2d9 | 172.20.15.61  | 172.20.15.62
(4 rows)

I was then able to create a qsfp1 link and give it the same address already in use by qsfp0:

% oxide system networking link add --rack $rack --switch switch1 --port qsfp1 --fec rs --speed 100
% oxide system hardware switch-port apply-settings --rack-id $rack --switch-slot switch1 --port qsfp1 --port-settings switch1-qsfp1
% oxide system networking addr add --rack $rack --switch switch1 --port qsfp1 --addr 172.20.15.62/29 --lot john-static

and now we have multiple entries for the same IP in different (applied) switch port settings groups:

root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from switch_port_settings;
                   id                  |      name       |         description          |         time_created          |         time_modified         | time_deleted
---------------------------------------+-----------------+------------------------------+-------------------------------+-------------------------------+---------------
  69e29fca-3312-4e55-8ed6-d91c39d1bc69 | switch1-qsfp1   | switch port settings         | 2026-08-05 18:05:22.236549+00 | 2026-08-05 18:05:22.236549+00 | NULL
  86e1ff51-74e8-4827-98b2-77897fb08104 | default-uplink0 | initial uplink configuration | 2026-08-04 21:30:45.776324+00 | 2026-08-04 21:30:45.776324+00 | NULL
  d1ae261e-1b64-4da2-8896-bf1318c55896 | default-uplink1 | initial uplink configuration | 2026-08-04 21:30:45.981783+00 | 2026-08-04 21:30:45.981783+00 | NULL
(3 rows)

root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from omicron.public.switch_port_settings_address_config;
            port_settings_id           |         address_lot_block_id         |      rsvd_address_lot_block_id       |     address     | interface_name | vlan_id
---------------------------------------+--------------------------------------+--------------------------------------+-----------------+----------------+----------
  69e29fca-3312-4e55-8ed6-d91c39d1bc69 | b202fea9-a857-4055-92b6-d460e3a6b43e | 19fb579a-7f02-4834-b81b-ce04dc18038a | 172.20.15.62/29 | phy0           | NULL
  86e1ff51-74e8-4827-98b2-77897fb08104 | df4c0a42-afd6-4392-a79e-2a0951f85c09 | 4cc7cc3a-7de5-4a86-bc32-bc9edf4f0927 | 172.20.15.61/29 | phy0           | NULL
  d1ae261e-1b64-4da2-8896-bf1318c55896 | df4c0a42-afd6-4392-a79e-2a0951f85c09 | b9cf0ed3-8c9c-40a8-a27b-d8bda8439fc7 | 172.20.15.62/29 | phy0           | NULL
(3 rows)

When the reconcilers on #10961 attempted to apply these settings to dpd, it kicked back a 409 conflict; the dpd logs contain:

18:45:23.400Z ERRO dpd: failed to add ipv4 address
    addr = 172.20.15.62
    error = ASIC error: Exists
    port = 324
18:45:23.400Z ERRO dpd: port_settings op failed: Switch(Exists)
18:45:23.400Z INFO dpd: request completed
    error_message_external = Conflict
    error_message_internal = Conflict
    latency_us = 284
    local_addr = [fde0:9b6b:5940:102::2]:12224
    method = POST
    remote_addr = [fde0:9b6b:5940:102::1]:59130
    req_id = f72ff43e-3739-4918-ab4f-127acef45b1c
    response_code = 409
    server_id = 1
    unit = api-server
    uri = /port/qsfp1/settings?tag=omicron

We should be rejecting this before it makes it down to an internal dpd conflict, and presumably we shouldn't allow overlapping address lots either?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the external networking address-lot and address-add API entry points, then trace how switch port settings are assembled and sent to the reconcilers and dpd. Reproduce the address-lot overlap and duplicate switch-port address cases from the issue; done means conflicting lots or applied addresses are rejected before dpd returns a 409 conflict.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.