thermal/transceivers: QSFP temperature may not be reliable

We have observed QSFP transceivers reporting temperatures that have been hot enough to cause an immediate power-down of the sidecar as part of the thermal monitoring task.

However, we have also now seen cases where the reported temperatures are suspicious, for example:

  23 1470        7     4917 ControlPwm(0x0)
  24 1582        7        1 PowerDownDueTo { sensor_id: SensorId(0x4a), worst_case_temp: Celsius(123.08184) }
  25 1586        7        1 LastRealTemperature { sensor_id: SensorId(0x4a), temperature: Celsius(122.83984), age_s: 0.484 }
  26 1603        7        1 AutoState(Uncontrollable)
  27 1474        7        1 PowerDownAt(0x3aaa0f10)
  28 1247        7        1 PowerModeChanged(PowerBitmask(0b1))
  29 1102        7        1 AutoState(Boot)
  30 1506        7        1 AutoState(Running)
  31 1470        7    65535 ControlPwm(0x0)

In this case, the system was at 0% fans one second, meaning that the transceivers are all under their nominal 65C limit. The next second, we power down due to a transceiver reporting 122.8C. The next second, we are back in Running mode with the fans commanded at 0%, which means the transceiver was again under 65C.

It seems extremely unlikely that the QSFP would experience a 60C excursion over a few seconds.

We have also seen some reports of transceivers reporting negative temperatures, which are again extremely unlikely in a climate controlled datacenter.

For these reasons, we may want to modify transceivers-server to report or reject potential "glitches". Mechanisms proposed so far include:

1. ALWAYS poll each temperature sensor twice in a row, if the readings are outside some reasonable span (5C or so), reject both samples and try again on the next tick.
2. If a temperature sensor reports a suspicious value, THEN repoll. This could be:
   1. If one transceiver is significantly different than other ones (if present)
   2. If the temperature jumps more than some amount between 1hz samples (some kind of moving average?)
   3. Some more intelligent filter, like using a Kalman filter and monitoring the sample variance
3. Poll temperature sensors more often (currently 1Hz), but potentially report less often, discarding any suspicious samples

We may want to avoid pre-filtering/averaging potentially bad data: a very large bad-sample could distort the average, and will always introduce additional lag on top of the filtering delay in the thermal PID loop.

Start in transceivers-server by tracing the thermal monitoring task and its current 1 Hz temperature polling. Compare the proposed repolling, rejection, and filtering approaches with the existing power-down behavior. Done means a selected glitch-handling policy is implemented and its behavior is covered by tests for suspicious temperature readings.