oxidecomputer / oxidecomputer/maghemite

rib_priority should be an option

Open
#646 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
94
Forks
6
Avg merge
1d 8h
Merged PRs (30d)
12

Description

In #359 local_pref was renamed to rib_priority and went from an Option<u32> to a u8. This has cause a misalignment with Omicron where route priorities are optional.

When nexus has a route to reconcile to maghemite that has no specified priority, since rib_priority is not optional it forces Nexus into choosing a sentinel when sending routes to maghemite. Then on the next reconciler run, nexus sees that the route it has in the database has no priority, but the route that it sees form maghemite has a priority, so it deletes the route and recreates it. We see this in the following logs

Deletes:

21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): deleting static routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:457
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    routes = {Switch1: DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }, Switch0: DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }}
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): removing static routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2343
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    request = DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
    switch_location = Switch1
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): removing static routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2343
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    request = DeleteStaticRouteRequest { v4: DeleteStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: DeleteStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
    switch_location = Switch0

Recreates:

21:53:39.578Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): adding static routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:463
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    routes = {Switch0: AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }, Switch1: AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }}
21:53:39.578Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): adding static routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2388
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    request = AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
    switch_location = Switch0
21:53:39.579Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): adding static routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:2388
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    request = AddStaticRouteRequest { v4: AddStaticRoute4Request { routes: StaticRoute4List { list: [StaticRoute4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, rib_priority: 1, vlan_id: None }] } }, v6: AddStaticRoute6Request { routes: StaticRoute6List { list: [] } } }
    switch_location = Switch1

Where the difference is in the priority

21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): retrieved existing routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:433
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    routes = {Switch1: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: Some(1) })}, Switch0: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: Some(1) })}}
21:53:39.577Z INFO 1a90c686-13b4-4d8f-8ecb-7f6bd4a040cf (ServerContext): retrieved desired routes
    background_task = switch_port_config_manager
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:437
    rack_id = 73cdd12c-a64a-43cc-a5c0-12c3add70029
    routes = {Switch1: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: None })}, Switch0: {V4(SwitchStaticRouteV4 { nexthop: 172.20.15.65, prefix: Prefix4 { value: 0.0.0.0, length: 0 }, vlan: None, priority: None })}}

While this could be fixed in nexus to recognize extended equality over sentinel values, that seems quite fraught. We should have maghemite line up with omicron in terms of having route priorities be optional.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing maghemite's static-route representation and the AddStaticRouteRequest/DeleteStaticRouteRequest handling related to rib_priority. Compare that behavior with the optional priority shown in Nexus's sync_switch_configuration.rs logs. Done means routes without a specified priority remain optional across maghemite and no longer trigger delete-and-recreate reconciliation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.