Tracking Issue: "Hot Transceivers"

There has been a lot of discussion in the past few days (after a lot of discussion over the last long while!) of how to deal with a single hot transceiver from shutting down the entire sidecar.

This pathology is observed somewhat regularly, both in manufacturing as well as customer support tickets. After looking at some previous reports, it is also possible that the transceivers are not getting "too hot" at all, and the cause of the shutdowns we have seen are actually just a "glitch" when reading temperature over I2C, as we have seen sudden jumps from <60C to >120C and back to <60C in the span of a few seconds, as well as at least one case where negative temperatures were observed.

As of today, the following items are up for discussion. I will try to link these to existing issues where possible (and make them subissues), and cross-link to this tracking issue where ever possible.

From discussions with @hawkw and @bnaecker today, my suggestions are:

Okay, so collecting some dots:

1. We want the thermal task's logging to be smarter - https://github.com/oxidecomputer/hubris/issues/2662
2. We probably don't want a single bad xcvr temp reading to kill the sidecar. We have a few choices here:
   1. Make REALLY sure too-hot-xcvrs are too-hot, do filtering or glitch-rejection of some kind - #2664
   2. Never let xcvrs kill the system, let them ramp up the fan but not kill the system (TODO: open issue for this)
      • Old PR partially doing this: https://github.com/oxidecomputer/hubris/pull/2374
   3. Figure out how to cut the power to one xcvr, without cutting the whole system - https://github.com/oxidecomputer/hubris/issues/2355.
3. When we DO get too hot, we don't currently shut down the Front IO panel and/or transceivers, only the tofino.
   1. We probably have the authority to kill the transceivers (and maybe the front-io panel) if we DO sequence to A2 (either for overheat, or just by command), because the tofino is down and the host no longer has control (TODO: open issue for this)
   2. We need to make sure we can come back from this state later when we are sequenced back to A0. (TODO: open issue for this)

I'll fill in details that were discussed in each of the subissues. I'll keep this tracking issue for coordinating which efforts we want to focus on first.

Items 1 and 2.1 are probably first on the list, so we can get better data when this does happen.

Start by reading the linked issues #2662, #2664, and #2355, along with PR #2374, to understand the proposed thermal-handling options and existing work. This tracking issue is done when the preferred efforts are agreed, the necessary follow-up issues are opened, and their coordination is clear.