SP not talking on one network port

When updating the colo rack, we noticed that we couldn't talk to the SP in cubby 2 from scrimlet 14 / switch 0.

root@oxz_switch0:~# ping fe80::aa40:25ff:fe04:24d
ICMPv6 Address Unreachable from gateway fe80::aa40:25ff:fe05:1502%gimlet25
 for icmp6 from fe80::aa40:25ff:fe05:1502 to fe80::aa40:25ff:fe04:24d
ICMPv6 Address Unreachable from gateway fe80::aa40:25ff:fe05:1502%gimlet25
 for icmp6 from fe80::aa40:25ff:fe05:1502 to fe80::aa40:25ff:fe04:24d
ICMPv6 Address Unreachable from gateway fe80::aa40:25ff:fe05:1502%gimlet25
 for icmp6 from fe80::aa40:25ff:fe05:1502 to fe80::aa40:25ff:fe04:24d

We can talk to it through scrimlet 16 / switch 1.

Monorail reports that the link is down, which is unusual:

PortStatus(Ok(PortStatus { port: 15, cfg: PortConfig { mode: Sgmii(Speed100M), dev: (Dev2g5, 7), serdes: (Serdes6g, 7) }, link_status: Error, phy_status: None, counters: PortCounters { rx: PacketCount { multicast: 0, unicast: 0, broadcast: 0 }, tx: PacketCount { multicast: 694, unicast: 0, broadcast: 0 }, link_down_sticky: true, phy_link_down_sticky: false } }))

This is a direct SGMII connection from the VSC7448, so LinkStatus::Error indicates that either signal detect or sync status is bad:

if reg.link_status() == 0 {
    LinkStatus::Down
} else if reg.signal_detect() == 0 || reg.sync_status() == 0 {
    LinkStatus::Error
} else {
    LinkStatus::Up
}

• SP dump from the Gimlet is in /staff/rack3/BRM42220001/2025-06-11
• SP dump from the Sidecar is in /staff/rack3/BRM44220010/2025-06-11

There's nothing particularly exciting in either dump, unfortunately.

Resetting the switch with faux-mgs --interface sidecar0 component-details monorail does not recover the link, which suggests that the issue is on the Gimlet SP's side. Resetting the Gimlet SP also doesn't recover the link, but changes the port status to LinkStatus::Down. Power-cycling the sled through Ignition also doesn't bring this link back (!!)

See also https://github.com/oxidecomputer/colo/issues/108 for other issues with this sled.

Our disposition is to pull the sled and bring it back to the lab for analysis.

(of course, I'm sure that the network will start working again under controlled circumstances)

Start with the Monorail port-status output and the Gimlet and Sidecar dumps in /staff/rack3/BRM42220001/2025-06-11 and /staff/rack3/BRM44220010/2025-06-11. Review the VSC7448 SGMII LinkStatus logic and compare with colo issue 108, then investigate the sled in the lab after it is pulled. Done means identifying and documenting the cause of the failed link.