Switch port settings become empty/contradictory after online update

Lab rack madrid was online updated to 76aa5fc0607b514c2376a99694fc68cb2d758616 following mupdate (without prior completion of recovery flagged from a3fee0ec59cc2beca911d13e3dc5a58dffbed8db). Recovery was completed appropriately on the prior version, and online update restarted.

Following the online update, sled 14's sled-agent goes into maintenance with:

12:36:19.721Z INFO SledAgent (ConfigReconcilerTask): obtained file source for zone, going to start it
    file = sled-agent/config-reconciler/src/reconciler_task/zones.rs:387
    file_source = OmicronResolvableFileSource { location: Artifact { hash: Ok(ArtifactHash("146af284a9ccad05562e3b466977cf4ac30cc1b1fb6e4404e43ee934f639820b")) }, file_source: ResolvableFileSource { file_name: "146af284a9ccad05562e3b466977cf4ac30cc1b1fb6e4404e43ee934f639820b", search_paths: ["/pool/int/d1268a8d-f6db-48db-9a6a-784b386a129e/update", "/pool/int/9a539a0d-6e57-4cb6-8813-1f23a1920f3f/update"] } }
    zone_name = oxz_cockroachdb_f8ce44a7-bd51-4d5d-b74f-dbc6441211da

thread 'tokio-rt-worker' (127) panicked at sled-agent/early-networking/src/lib.rs:228:9:
assertion failed: !uplinked_switch_zone_addrs.is_empty()
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ Jun 18 12:36:20 Stopping because all processes in service exited. ]
[ Jun 18 12:36:20 Executing stop method (:kill). ]
[ Jun 18 12:36:20 Restarting too quickly, changing state to maintenance. ]

Investigation by @jgallagher shows contradictory issues on sled 17's nexus logs.

BRM42220004 # grep 'updating bootstore config' /pool/ext/30007ebe-e206-42b6-8ecc-9d2a8f81d406/crypt/debug/oxz_nexus_a830bef8-82f4-4549-9187-89fa409d378d/oxide-nexus:default.log.1781781298 | looker -o long
2026-06-18 07:38:23.715Z INFO a830bef8-82f4-4549-9187-89fa409d378d/13192 (ServerContext) on oxz_nexus_a830bef8-82f4-4549-9187-89fa409d378d: updating bootstore config
    background_task = switch_port_config_manager
    config = SystemNetworkingConfig { rack_network_config: RackNetworkConfig { rack_subnet: Ipv6Net { addr: fd2a:6309:82b3:100::, width: 56 }, infra_ip_first: 172.20.15.37, infra_ip_last: 172.20.15.38, ports: [PortConfig { routes: [], addresses: [UplinkAddressConfig { address: Static { ip_net: UplinkIpNet(V4(Ipv4Net { addr: 172.20.15.37, width: 29 })) }, vlan_id: None }], switch: switch0, port: "qsfp0", uplink_port_speed: Speed100G, uplink_port_fec: Some(Rs), bgp_peers: [BgpPeerConfig { asn: 65103, port: "qsfp0", addr: Numbered { ip: RouterPeerIpAddr(172.20.15.35) }, hold_time: Some(6), idle_hold_time: Some(3), delay_open: Some(0), connect_retry: Some(3), keepalive: Some(2), remote_asn: None, min_ttl: None, md5_auth_key: None, multi_exit_discriminator: None, communities: [], local_pref: None, enforce_first_as: false, allowed_import: NoFiltering, allowed_export: NoFiltering, vlan_id: None }], autoneg: false, lldp: Some(LldpPortConfig { status: Enabled, chassis_id: None, port_id: Some("switch 0 qsfp0/0"), port_description: None, system_name: Some("madrid"), system_description: None, management_addrs: None }), tx_eq: None }], bgp: [BgpConfig { asn: 65103, originate: [V4(Ipv4Net { addr: 172.20.35.0, width: 24 })], shaper: None, checker: None, max_paths: MaxPathConfig(1) }], bfd: [] }, blueprint_external_networking_config: Some(BlueprintExternalNetworkingConfig { blueprint_external_networking_generation: Generation(1), service_zone_nat_entries: ServiceZoneNatEntries({4a796334-ee5c-4fa1-b363-9da3dfe2ec05 (omicron_zone): ServiceZoneNatEntry { zone_id: 4a796334-ee5c-4fa1-b363-9da3dfe2ec05 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:102::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 77])), vni: Vni(100), kind: ExternalDns { external_ip: 172.20.35.2 } }, 8d392f55-447c-44a7-af65-15e0c0d7f427 (omicron_zone): ServiceZoneNatEntry { zone_id: 8d392f55-447c-44a7-af65-15e0c0d7f427 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:103::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 224])), vni: Vni(100), kind: ExternalDns { external_ip: 172.20.35.3 } }, a830bef8-82f4-4549-9187-89fa409d378d (omicron_zone): ServiceZoneNatEntry { zone_id: a830bef8-82f4-4549-9187-89fa409d378d (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:101::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 184, 154])), vni: Vni(100), kind: Nexus { external_ip: 172.20.35.5 } }, ae2bc284-e25d-4e8f-bf98-a211e49e3dd5 (omicron_zone): ServiceZoneNatEntry { zone_id: ae2bc284-e25d-4e8f-bf98-a211e49e3dd5 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:104::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 77])), vni: Vni(100), kind: Nexus { external_ip: 172.20.35.4 } }, b275f6a1-84b2-412b-9900-2d9a42617aa3 (omicron_zone): ServiceZoneNatEntry { zone_id: b275f6a1-84b2-412b-9900-2d9a42617aa3 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:101::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 140])), vni: Vni(100), kind: ExternalDns { external_ip: 172.20.35.1 } }, cae7387d-bbfc-4a57-a0d0-c9c8aed49de2 (omicron_zone): ServiceZoneNatEntry { zone_id: cae7387d-bbfc-4a57-a0d0-c9c8aed49de2 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:102::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 46])), vni: Vni(100), kind: BoundaryNtp { snat_cfg: SourceNatConfig { ip: 172.20.35.7, first_port: 16384, last_port: 32767 } } }, dffd1ff5-48a7-4574-8964-c825e29e17bf (omicron_zone): ServiceZoneNatEntry { zone_id: dffd1ff5-48a7-4574-8964-c825e29e17bf (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:102::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 65])), vni: Vni(100), kind: Nexus { external_ip: 172.20.35.6 } }, eac0afbf-18aa-4e77-9ab8-434eb5304214 (omicron_zone): ServiceZoneNatEntry { zone_id: eac0afbf-18aa-4e77-9ab8-434eb5304214 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:101::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 81])), vni: Vni(100), kind: BoundaryNtp { snat_cfg: SourceNatConfig { ip: 172.20.35.7, first_port: 0, last_port: 16383 } } }}) }) }
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1480
    rack_id = bc81f198-6a70-4d35-b333-6fe39682a747
2026-06-18 07:39:09.109Z INFO a830bef8-82f4-4549-9187-89fa409d378d/13192 (ServerContext) on oxz_nexus_a830bef8-82f4-4549-9187-89fa409d378d: updating bootstore config
    background_task = switch_port_config_manager
    config = SystemNetworkingConfig { rack_network_config: RackNetworkConfig { rack_subnet: Ipv6Net { addr: fd2a:6309:82b3:100::, width: 56 }, infra_ip_first: 172.20.15.37, infra_ip_last: 172.20.15.38, ports: [], bgp: [BgpConfig { asn: 65103, originate: [V4(Ipv4Net { addr: 172.20.35.0, width: 24 })], shaper: None, checker: None, max_paths: MaxPathConfig(1) }], bfd: [] }, blueprint_external_networking_config: Some(BlueprintExternalNetworkingConfig { blueprint_external_networking_generation: Generation(1), service_zone_nat_entries: ServiceZoneNatEntries({4a796334-ee5c-4fa1-b363-9da3dfe2ec05 (omicron_zone): ServiceZoneNatEntry { zone_id: 4a796334-ee5c-4fa1-b363-9da3dfe2ec05 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:102::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 77])), vni: Vni(100), kind: ExternalDns { external_ip: 172.20.35.2 } }, 8d392f55-447c-44a7-af65-15e0c0d7f427 (omicron_zone): ServiceZoneNatEntry { zone_id: 8d392f55-447c-44a7-af65-15e0c0d7f427 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:103::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 224])), vni: Vni(100), kind: ExternalDns { external_ip: 172.20.35.3 } }, a830bef8-82f4-4549-9187-89fa409d378d (omicron_zone): ServiceZoneNatEntry { zone_id: a830bef8-82f4-4549-9187-89fa409d378d (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:101::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 184, 154])), vni: Vni(100), kind: Nexus { external_ip: 172.20.35.5 } }, ae2bc284-e25d-4e8f-bf98-a211e49e3dd5 (omicron_zone): ServiceZoneNatEntry { zone_id: ae2bc284-e25d-4e8f-bf98-a211e49e3dd5 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:104::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 135, 77])), vni: Vni(100), kind: Nexus { external_ip: 172.20.35.4 } }, b275f6a1-84b2-412b-9900-2d9a42617aa3 (omicron_zone): ServiceZoneNatEntry { zone_id: b275f6a1-84b2-412b-9900-2d9a42617aa3 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:101::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 140])), vni: Vni(100), kind: ExternalDns { external_ip: 172.20.35.1 } }, cae7387d-bbfc-4a57-a0d0-c9c8aed49de2 (omicron_zone): ServiceZoneNatEntry { zone_id: cae7387d-bbfc-4a57-a0d0-c9c8aed49de2 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:102::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 46])), vni: Vni(100), kind: BoundaryNtp { snat_cfg: SourceNatConfig { ip: 172.20.35.7, first_port: 16384, last_port: 32767 } } }, dffd1ff5-48a7-4574-8964-c825e29e17bf (omicron_zone): ServiceZoneNatEntry { zone_id: dffd1ff5-48a7-4574-8964-c825e29e17bf (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:102::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 234, 65])), vni: Vni(100), kind: Nexus { external_ip: 172.20.35.6 } }, eac0afbf-18aa-4e77-9ab8-434eb5304214 (omicron_zone): ServiceZoneNatEntry { zone_id: eac0afbf-18aa-4e77-9ab8-434eb5304214 (omicron_zone), sled_underlay_ip: fd2a:6309:82b3:101::1, nic_mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 81])), vni: Vni(100), kind: BoundaryNtp { snat_cfg: SourceNatConfig { ip: 172.20.35.7, first_port: 0, last_port: 16383 } } }}) }) }
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1480
    rack_id = bc81f198-6a70-4d35-b333-6fe39682a747

The full logs can be found at /staff/skaram/madrid-sleds-nexus/