Diagnostic features for update failures

We had a sidecar SP fail update at a customer site today in a rather ambiguous manner. This issue is intended to collect ideas for diagnostic tools we could have built that would have helped today, so that we can hopefully build them before this reproduces much more.

One possibility is that this is simply an MGS timeout that has drifted out of sync with how long Sidecar takes to boot in practice. We know Sidecar boot is nondeterministic (https://github.com/oxidecomputer/hardware-sidecar/issues/741) so if the timeout is marginal, it could happen rarely for certain units.

Potential root causes I've floated, and tools that might help distinguish them, include:

• Update was written incorrectly due to corruption, off-by-one, or other bug.
  • The ability to read out, or at least hash, the idle bank would let us check for this.
• SP rebooted into new image which was written correctly, but something failed during initialization that prevented the network from coming up.
  • It would be really great to be able to read a dump from a previous boot out of the dump area, to see if anything panicked last boot.
• Sidecar may have taken too long to start up for the timeout in MGS, and this might all be an illusion.
  • MGS may want to revise up that timeout (I would also argue for making it configurable, for the next time this happens)
  • We should take a pass over Sidecar startup and check for any optimizations we could make there.

Please add more ideas.

Start with the issue's proposed diagnostic areas: MGS update timeouts, Sidecar startup, idle-bank reads or hashes, and previous-boot dump retrieval. The payload names no files, tests, or entry points, and does not define a completion condition; first turn the ideas into a scoped design with an agreed diagnostic and success criteria.