oxidecomputer / oxidecomputer/omicron
External networking API allows addition of conflicting address lots and IP addresses
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
In testing #10961, I made some networking changes to dublin that Nexus allowed but I think should not have. I initially set dublin up using BGP with qsfp0 configured on both switches:
[rack_network_config]
infra_ip_first = "172.20.15.61"
infra_ip_last = "172.20.15.62"
[[rack_network_config.bgp]]
asn = 65106
originate = ["172.20.29.0/24","172.20.36.0/24"]
[rack_network_config.switch0.qsfp0]
routes = []
addresses = [{address = "172.20.15.61/29"}]
uplink_port_speed = "100G"
uplink_port_fec = "rs"
autoneg = false
bgp_peers = [{asn = 65106, addr = "172.20.15.59", port = "qsfp0"}]
[rack_network_config.switch1.qsfp0]
routes = []
addresses = [{address = "172.20.15.62/29"}]
uplink_port_speed = "100G"
uplink_port_fec = "rs"
autoneg = false
bgp_peers = [{asn = 65106, addr = "172.20.15.59", port = "qsfp0"}]
I wanted to try adding a static address / route to switch1/qsfp1, and stumbled forward blindly using some configs that accidentally overlapped with the above, but Nexus allowed me to set up things that it seems like it should not have. When I initially tried to add the address, I got an address unavailable, which is perfectly reasonable:
% oxide system networking addr add --rack $rack --switch switch1 --port qsfp0 --addr 172.20.15.62/29 --lot initial-infra
Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "c5fc8ff4-7f18-4c0f-a4a4-06bcedd0c731", "content-length": "128", "date": "Wed, 05 Aug 2026 18:00:40 GMT"}; value: Error { error_code: Some("InvalidRequest"), message: "address unavailable", request_id: "c5fc8ff4-7f18-4c0f-a4a4-06bcedd0c731" }
However, I was able to create a different address lot with the same addresses:
% cat add-static-routing.json
{
"name": "john-static",
"description": "address lot for static routing",
"kind": "infra",
"blocks": [
{
"first_address": "172.20.15.61",
"last_address": "172.20.15.62"
}
]
}
% oxide system networking address-lot create --name john-static --description 'adding static routing' --json-body add-static-routing.json
{
"blocks": [
{
"first_address": "172.20.15.61",
"id": "b202fea9-a857-4055-92b6-d460e3a6b43e",
"last_address": "172.20.15.62"
}
],
"lot": {
"description": "address lot for static routing",
"id": "eb537eab-0e08-4fa8-8eca-263165cd1ec7",
"kind": "infra",
"name": "john-static",
"time_created": "2026-08-05T18:01:58.478720Z",
"time_modified": "2026-08-05T18:01:58.478720Z"
}
}
This already seems bogus, because we have two different address lots covering the same IPs in CRDB now:
root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from address_lot;
id | name | description | time_created | time_modified | time_deleted | kind
---------------------------------------+---------------+------------------------------------------+-------------------------------+-------------------------------+--------------+--------
eb537eab-0e08-4fa8-8eca-263165cd1ec7 | john-static | address lot for static routing | 2026-08-05 18:01:58.47872+00 | 2026-08-05 18:01:58.47872+00 | NULL | infra
f14e49a5-4299-445e-9853-17614778b2d9 | initial-infra | initial infrastructure ip address lot | 2026-08-04 21:30:45.603884+00 | 2026-08-04 21:30:45.603884+00 | NULL | infra
f5041974-fb81-48b7-8124-25dbee726713 | as65106-lot | Address lot for announce set in as 65106 | 2026-08-04 21:30:45.662304+00 | 2026-08-04 21:30:45.662304+00 | NULL | infra
(3 rows)
root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from address_lot_block;
id | address_lot_id | first_address | last_address
---------------------------------------+--------------------------------------+---------------+----------------
1b2b84c0-1c31-4538-bf3e-bc605fe655b6 | f5041974-fb81-48b7-8124-25dbee726713 | 172.20.36.0 | 172.20.36.255
56625a05-357c-41ef-a4b4-a90e81432167 | f5041974-fb81-48b7-8124-25dbee726713 | 172.20.29.0 | 172.20.29.255
b202fea9-a857-4055-92b6-d460e3a6b43e | eb537eab-0e08-4fa8-8eca-263165cd1ec7 | 172.20.15.61 | 172.20.15.62
df4c0a42-afd6-4392-a79e-2a0951f85c09 | f14e49a5-4299-445e-9853-17614778b2d9 | 172.20.15.61 | 172.20.15.62
(4 rows)
I was then able to create a qsfp1 link and give it the same address already in use by qsfp0:
% oxide system networking link add --rack $rack --switch switch1 --port qsfp1 --fec rs --speed 100
% oxide system hardware switch-port apply-settings --rack-id $rack --switch-slot switch1 --port qsfp1 --port-settings switch1-qsfp1
% oxide system networking addr add --rack $rack --switch switch1 --port qsfp1 --addr 172.20.15.62/29 --lot john-static
and now we have multiple entries for the same IP in different (applied) switch port settings groups:
root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from switch_port_settings;
id | name | description | time_created | time_modified | time_deleted
---------------------------------------+-----------------+------------------------------+-------------------------------+-------------------------------+---------------
69e29fca-3312-4e55-8ed6-d91c39d1bc69 | switch1-qsfp1 | switch port settings | 2026-08-05 18:05:22.236549+00 | 2026-08-05 18:05:22.236549+00 | NULL
86e1ff51-74e8-4827-98b2-77897fb08104 | default-uplink0 | initial uplink configuration | 2026-08-04 21:30:45.776324+00 | 2026-08-04 21:30:45.776324+00 | NULL
d1ae261e-1b64-4da2-8896-bf1318c55896 | default-uplink1 | initial uplink configuration | 2026-08-04 21:30:45.981783+00 | 2026-08-04 21:30:45.981783+00 | NULL
(3 rows)
root@[fde0:9b6b:5940:101::3]:32221/omicron> select * from omicron.public.switch_port_settings_address_config;
port_settings_id | address_lot_block_id | rsvd_address_lot_block_id | address | interface_name | vlan_id
---------------------------------------+--------------------------------------+--------------------------------------+-----------------+----------------+----------
69e29fca-3312-4e55-8ed6-d91c39d1bc69 | b202fea9-a857-4055-92b6-d460e3a6b43e | 19fb579a-7f02-4834-b81b-ce04dc18038a | 172.20.15.62/29 | phy0 | NULL
86e1ff51-74e8-4827-98b2-77897fb08104 | df4c0a42-afd6-4392-a79e-2a0951f85c09 | 4cc7cc3a-7de5-4a86-bc32-bc9edf4f0927 | 172.20.15.61/29 | phy0 | NULL
d1ae261e-1b64-4da2-8896-bf1318c55896 | df4c0a42-afd6-4392-a79e-2a0951f85c09 | b9cf0ed3-8c9c-40a8-a27b-d8bda8439fc7 | 172.20.15.62/29 | phy0 | NULL
(3 rows)
When the reconcilers on #10961 attempted to apply these settings to dpd, it kicked back a 409 conflict; the dpd logs contain:
18:45:23.400Z ERRO dpd: failed to add ipv4 address
addr = 172.20.15.62
error = ASIC error: Exists
port = 324
18:45:23.400Z ERRO dpd: port_settings op failed: Switch(Exists)
18:45:23.400Z INFO dpd: request completed
error_message_external = Conflict
error_message_internal = Conflict
latency_us = 284
local_addr = [fde0:9b6b:5940:102::2]:12224
method = POST
remote_addr = [fde0:9b6b:5940:102::1]:59130
req_id = f72ff43e-3739-4918-ab4f-127acef45b1c
response_code = 409
server_id = 1
unit = api-server
uri = /port/qsfp1/settings?tag=omicron
We should be rejecting this before it makes it down to an internal dpd conflict, and presumably we shouldn't allow overlapping address lots either?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the external networking address-lot and address-add API entry points, then trace how switch port settings are assembled and sent to the reconcilers and dpd. Reproduce the address-lot overlap and duplicate switch-port address cases from the issue; done means conflicting lots or applied addresses are rejected before dpd returns a 409 conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100