wicket may fail to show "Downloading installinator" progress if the other scrimlet is serving the same host image

If we follow this sequence:

1. Perform a mupdate via one scrimlet (call this "scrimlet 14")
2. Leave scrimlet 14 (in particular, its switch zone) untouched - MGS will continue to hold the phase 2 recovery OS image in-memory to serve to any SP that requests it
3. Begin a mupdate that uses the identical recovery OS image via the other scrimlet ("scrimlet 16")

it's possible that the mupdate in step 3 will show no progress information during the "Downloading installinator, waiting for it to start step", even though installinator is actually being downloaded successfully.

As of the beginning of the mupdate via scrimlet 16 in step 3, the MGS instances on both scrimlets have the installinator OS image and are able to serve it to any SP that asks for it over the management network. When the SP fetches data from MGS, it sends a multicast packet and will "attach" to whichever MGS instance it finds first, unless that instance stops responding (in which case it will start looking for either MGS instance again).

In step 3, the wicketd running on scrimlet 16 is asking its local MGS (literally via localhost) for progress information. But if the SP of the sled being updated attaches to the MGS on scrimlet 14, there will be no progress information available.

Today while in this situation on the dogfood rack, we shutdown the MGS service (via svcadm disable mgs) on scrimlet 14, and more or less immediately started seeing progress, because the SP flipped over and attached to the MGS on scrimlet 16. Re-enabling the service on scrimlet 14 did not affect continued progress messages, as the SP stayed attached to the scrimlet 16 MGS.

I'm not sure whether or not there's really a bug here: in normal operation I think we would not expect both MGS instances to be serving identical host images at the same time. If we decide we do want this situation to work and show progress, it may be tricky: wicketd has no way to talk to MGS on the other scrimlet (MGS only listens on localhost and, once the control plane is up, the underlay network, and wicketd only talks on localhost and the bootstrap network).

Start with the linked hubris task/control-plane-agent/src/mgs_gimlet/host_phase2.rs code and trace how an SP attaches to MGS during the reported two-scrimlet sequence. Reproduce the case with identical recovery OS images, then determine whether wicketd should expose progress when the SP attaches to the other MGS; done means the expected behavior is established and verified.