after miscabling, omicron setup and cold boot hung waiting for wrong switch info

I ran into trouble setting up Omicron on "madrid". This issue involves some misconfiguration and physical miscabling but I'm filing this in hopes that Omicron could deal with this better than hanging.

madrid is a 4-gimlet + 1-sidecar system, recently moved into a racklette. I went to deploy a custom build of Omicron on it (from branch https://github.com/oxidecomputer/omicron/pull/4466 but I don't think that matters here). It hung during setup. After about 15 minutes RSS seemed stuck here:

# grep -i rss $(svcs -L sled-agent) | looker | tail 
    addrs = {Switch0: fd00:1122:3344:103::2}
    file = sled-agent/src/bootstrap/early_networking.rs:187
04:14:15.361Z INFO SledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { version: Generation(3), zones: [OmicronZoneConfig { id: df0897f3-87da-401c-a2d2-fb0872072c5d, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d77ef109-9083-4ed9-b5b4-c01a4349b775, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 162a487a-9306-4278-a629-a09706dd0833, underlay_address: fd00:1122:3344:101::e, zone_type: InternalNtp { address: [fd00:1122:3344:101::e]:123, ntp_servers: ["5ff52d9e-91d3-4022-99c0-758213e0f085.host.control-plane.oxide.internal", "3de57298-66b6-4a7b-bb35-5ac378107241.host.control-plane.oxide.internal"], dns_servers: [fd00:1122:3344:1::1, fd00:1122:3344:2::1, fd00:1122:3344:3::1], domain: None } }] }
    file = sled-agent/src/rack_setup/service.rs:288
04:14:15.407Z INFO SledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { version: Generation(3), zones: [OmicronZoneConfig { id: 76460784-22cb-48d8-9cf6-504a367805ee, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 76f8adbc-d793-47c3-bdc0-158247110fb1, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5ff52d9e-91d3-4022-99c0-758213e0f085, underlay_address: fd00:1122:3344:104::11, zone_type: BoundaryNtp { address: [fd00:1122:3344:104::11]:123, ntp_servers: ["ntp.eng.oxide.computer"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 9bf0f3de-b38e-4185-8185-efc4f05933d1, kind: Service { id: 5ff52d9e-91d3-4022-99c0-758213e0f085 }, name: Name("ntp-5ff52d9e-91d3-4022-99c0-758213e0f085"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 232, 31])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 172.20.28.5, first_port: 0, last_port: 16383 } } }] }
    file = sled-agent/src/rack_setup/service.rs:288
04:14:15.447Z INFO SledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { version: Generation(3), zones: [OmicronZoneConfig { id: 7507d5d9-b4d7-4367-ae81-36c44810f619, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: daf7a7f1-2fdf-4c8f-b380-1456a860880f, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 3de57298-66b6-4a7b-bb35-5ac378107241, underlay_address: fd00:1122:3344:102::10, zone_type: BoundaryNtp { address: [fd00:1122:3344:102::10]:123, ntp_servers: ["ntp.eng.oxide.computer"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 29aca2de-059d-4f39-91fe-3b13b19ee96c, kind: Service { id: 3de57298-66b6-4a7b-bb35-5ac378107241 }, name: Name("ntp-3de57298-66b6-4a7b-bb35-5ac378107241"), ip: 172.30.3.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 142, 191])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 172.20.28.6, first_port: 16384, last_port: 32767 } } }] }
    file = sled-agent/src/rack_setup/service.rs:288
04:14:15.485Z INFO SledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { version: Generation(3), zones: [OmicronZoneConfig { id: a4ec3f4d-c9c1-4323-bfec-2b4d91fedca5, underlay_address: fd00:1122:3344:103::10, zone_type: InternalNtp { address: [fd00:1122:3344:103::10]:123, ntp_servers: ["5ff52d9e-91d3-4022-99c0-758213e0f085.host.control-plane.oxide.internal", "3de57298-66b6-4a7b-bb35-5ac378107241.host.control-plane.oxide.internal"], dns_servers: [fd00:1122:3344:1::1, fd00:1122:3344:2::1, fd00:1122:3344:3::1], domain: None } }] }
    file = sled-agent/src/rack_setup/service.rs:288

This code did definitely change in my branch but what it's doing is approximately the same as what would be happening on "main".

I took a look at sled 14, BRM42220081, and found that the ledger has generation 2 but we're trying to write generation 3. The internal DNS zone on this system appeared up and working. But from this sled's log, right after setting up DNS, we see:

00:49:01.661Z INFO SledAgent (ServiceManager): Resolving switch zone addresses in DNS
    file = sled-agent/src/bootstrap/early_networking.rs:253
00:49:01.739Z INFO SledAgent (ServiceManager): Querying MGS to determine switch location
    addr = fd00:1122:3344:103::2
    file = sled-agent/src/bootstrap/early_networking.rs:270
00:49:01.756Z INFO SledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:103::2
    file = sled-agent/src/bootstrap/early_networking.rs:295
00:49:01.756Z WARN SledAgent (ServiceManager): Failed to look up switch zone locations
    error = No switch locations found
    file = sled-agent/src/bootstrap/early_networking.rs:222
    retry_after = 131.223138ms

and this is happening in a loop, apparently indefinitely. I think this is part of the generation-3 request where we're provisioning a boundary NTP zone, entering opte_ports_needed(), and then getting into the early_networking code. What's important here is:

1. The code found switch 0. Indeed, MGS is reporting:

curl -i http://[fd00:1122:3344:103::2]:12225/local/switch-id
HTTP/1.1 200 OK
content-type: application/json
x-request-id: 2d08d9aa-e7bf-4f6c-8e92-3cb08193933c
content-length: 26
date: Sun, 28 Dec 1986 04:47:54 GMT

{"type":"switch","slot":0}

2. the code is looking for switch 1 because that's what's in my rss config file, in rack_network_config.ports[0].switch ("switch1").

So it looks like I just misconfigured it (that turned out to be not quite the case -- more on this below). So I tore everything down and took another lap with the config saying switch0. Setup worked. Great!

Then I did a cold boot (bounced all four Gimlets). At least one of them hung in exactly the same way, but this time:

00:18:02.148Z INFO SledAgent (ServiceManager): Resolving switch zone addresses in DNS
    file = sled-agent/src/bootstrap/early_networking.rs:253
00:18:02.187Z INFO SledAgent (ServiceManager): Querying MGS to determine switch location
    addr = fd00:1122:3344:103::2
    file = sled-agent/src/bootstrap/early_networking.rs:270
00:18:02.188Z INFO SledAgent (ServiceManager): Found Switch1 at fd00:1122:3344:103::2
    file = sled-agent/src/bootstrap/early_networking.rs:295
00:18:02.188Z WARN SledAgent (ServiceManager): Failed to look up switch zone locations
    error = No switch locations found
    file = sled-agent/src/bootstrap/early_networking.rs:222
    retry_after = 19.353309192s

Here it's finding switch 1 because that's what MGS reports:

{"type":"switch","slot":1}

but looking for switch 0 (based on what was in my config).

At this point @jgallagher took a look and discovered that the system was miscabled: see oxidecomputer/meta#324. For one of the Gimlets, the sole Sidecar was connected through the "switch 0" connector. For the other three, it was connected through the "switch 1" cable. MGS assumes all Gimlets will look the same in this regard and essentially picks what any of them says and then sticks to it -- hence why this was changing on this system.

Obviously something was deeply wrong on this system. But I wonder if we could better handle this kind of problem during setup and zone initialization? Ideas:

• When we kick off RSS from Wicket, could/should Wicket verify that the set of switches in the config matches what both MGS's report? In that case it could have told me this wasn't going to work before I tried to setup Omicron.
• When we go through the opte_ports_needed() path and try to look up a switch, if we expect one switch (based on the config) and we find one but it's the other switch, maybe we should stop immediately with an error?

Start by tracing the switch lookup in sled-agent/src/bootstrap/early_networking.rs, the Omicron zone setup path in sled-agent/src/rack_setup/service.rs, and the opte_ports_needed() entry point, using the issue's MGS responses and rack_network_config. Done should include a defined setup-time response when the configured switch differs from the switch reported by MGS, with coverage or validation for the mismatched and matching cases.