oxidecomputer / oxidecomputer/maghemite
rib_priority should be an option
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 94
- Forks
- 6
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 12
Description
In #359 local_pref was renamed to rib_priority and went from an Option<u32> to a u8. This has cause a misalignment with Omicron where route priorities are optional.
When nexus has a route to reconcile to maghemite that has no specified priority, since rib_priority is not optional it forces Nexus into choosing a sentinel when sending routes to maghemite. Then on the next reconciler run, nexus sees that the route it has in the database has no priority, but the route that it sees form maghemite has a priority, so it deletes the route and recreates it. We see this in the following logs
Deletes:
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): deleting static routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:457
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
routes = {Switch1: DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }, Switch0: DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }}
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): removing static routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2343
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
request = DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
switch_location = Switch1
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): removing static routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2343
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
request = DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
switch_location = Switch0
Recreates:
21:53:39.578Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): adding static routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:463
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
routes = {Switch0: AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }, Switch1: AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }}
21:53:39.578Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): adding static routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2388
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
request = AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
switch_location = Switch0
21:53:39.579Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): adding static routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2388
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
request = AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
switch_location = Switch1
Where the difference is in the priority
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): retrieved existing routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:433
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
routes = {Switch1: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: Some(1) })}, Switch0: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: Some(1) })}}
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): retrieved desired routes
background_task = switch_port_config_manager
file = nexus/src/app/background/tasks/sync_switch_configuration.rs:437
rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
routes = {Switch1: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: None })}, Switch0: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: None })}}
While this could be fixed in nexus to recognize extended equality over sentinel values, that seems quite fraught. We should have maghemite line up with omicron in terms of having route priorities be optional.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing maghemite's static-route representation and the AddStaticRouteRequest/DeleteStaticRouteRequest handling related to rib_priority. Compare that behavior with the optional priority shown in Nexus's sync_switch_configuration.rs logs. Done means routes without a specified priority remain optional across maghemite and no longer trigger delete-and-recreate reconciliation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100