psc/observer: implement FRAM blackbox

The PSC Rev-C and Observer hardware contain an FRAM part providing fast-write non-volatile memory. Currently, we do not use this actively.

Pulling from https://github.com/oxidecomputer/hubris/issues/874#issuecomment-2065176798:

Behavior that would be helpful right now that is not yet implemented includes:

• The ability to read the "what happened" information out of the rectifier, including, at minimum, the fault/alert information accessible over PMBus.
• Recording that information to the FRAM so that it is accessible across a power cycle, since it can't come out via the management network if the power supply is off.
• Control logic to at least attempt to recover the rectifier and turn power back on. (This is a rare case in the system where the microcontroller really truly does need to be responsible for this class of decision, because the Bigger Computers may not be powered on, due to the rectifier trip).

Information we'd probably need to gather to do this:

• Which pieces of data from the rectifier should be logged in the FRAM, and in particular, any information specific to a fault diagnosis that we may not already be sending to sensors.
• A decision on what the rectifier fault recovery and retry logic should be in a lights-out situation. (It may be different for a -lab image or something.)
• Info on how to do the recovery process on the rectifier.

Likely implementation chunks, not all of which are in this repo, seem like:

• Detection of rectifier faults in the PSC firmware and/or extension of existing detection to write more information into the FRAM.
• FRAM support in the PSC firmware (I don't see existing support, though they tend to be software-compatible with 24xx series I2C EEPROMs, so it's possible we're using it already with that driver)
• Rectifier recovery support in the PSC firmware.
• Extensions to control plane agent to report the availability of an FRAM blob up-stack, and to allow it to be extracted and then marked as unneeded.
• Extensions to the control plane to notice and collect said blob and put it somewhere we can get to it.

Previous Discussion

What to log

Which pieces of data from the rectifier should be logged in the FRAM, and in particular, any information specific to a fault diagnosis that we may not already be sending to sensors.

From @rmustacc:

The minimum viable piece here that is useful is going to be starting with the standard PMBus alerting related register, STATUS_WORD which then refers to the other registers:

• STATUS_VOUT
• STATUS_IOUT
• STATUS_INPUT
• STATUS_CML
• STATUS_TEMPERATURE
• STATUS_MFR_SPECIFIC
• STATUS_FANS_1_2

Note, STATUS_CML is used for a number of different failures and things like unsupported/invalid command/data is likely cases I wouldn't log for, where as the memory and processor failures I would. The device has two rails the primary 54.5V and the 12V standby. If we had to focus on only a single rail for some reason it would want to be the 54.5V, though it's possible that the others. Mostly these STATUS words to me are the most interesting thing we can log if I'm being picky as other data would hopefully end up in sensors and related.

The other thing for us to consider here is that for everything other than INPUT_UV_F there is actually a black box that the controller generates that contains this as well as the readings that triggered this. It can store up to 5 of these. Perhaps grabbing and storing this on a fault, is actually what we should consider rather than a manual set.

More recent discussions

There was some discussion with @nathanaelhuffman where we discussed keeping a ring-buffer of voltage levels as well to be able to determine cases where the input power was lost due to facility outages as well.

One thing to note is that the FRAM part contains 8KiB of space. We will likely need to be thrifty when it comes to how much historical data we are able to capture, and when we do or do not want to overwrite old data.

Fault Recovery/Retry Logic

A decision on what the rectifier fault recovery and retry logic should be in a lights-out situation. (It may be different for a -lab image or something.)

From @rmustacc:

So, I think the interesting thing to me is that the PSC will survive on its 12V standby even if the 54.5V main stays up. To me what this suggests that our starting point should not be on a per-rectifier basis, but if all 6 rectifiers go down, then we should pretty much probably try to always recover right now. I think that's not the worst starting point. Obviously there are a lot of different ways we can go over time.

From @isobering:

In the near term, the fault recovery procedure should probably be something like:

• Detect a fault (either by monitoring the PWR_OK_L signals or by detecting a PMBus ALERT signal and reading the fault)
• Assert a high level on SP_TO_PS_PSU_x_EN_L to disable the rectifier
• Dump whatever error information you want over PMBus and write it to the FRAM
• Wait a small integer number of seconds for transient power line stuff to clear
• Assert a low level on SP_TO_PS_PSU_x_EN_L to enable the rectifier
• If the rectifier is still faulted after a small integer number of attempts (say, three attempts), turn the rectifier off and do not keep trying to recover - there's probably something wrong with the rectifier.

In the long term, I think the fault recovery behavior should be operator configurable, and it would be good to give them three options for recovering from rectifier faults:

• Keep the rectifier off and do not retry
• Retry [n] times, where the operator specifies the number of attempts [n]
• Retry continuously every [t] seconds, where the operator specifies the time [t]