want a way for Omicron code to report when supposedly impossible situations happen

It would be really handy to have some kind of interface either in Nexus or in Omicron at-large to report a situation that we thought was practically impossible has happened. Here are some examples:
https://github.com/oxidecomputer/omicron/blob/8d7f973204c6fbe22a04ee0136eee9f475f34a57/nexus/db-queries/src/db/datastore/deployment.rs#L1380-L1392
https://github.com/oxidecomputer/omicron/blob/8d7f973204c6fbe22a04ee0136eee9f475f34a57/nexus/reconfigurator/planning/src/mgs_updates/rot_bootloader.rs#L150-L157
https://github.com/oxidecomputer/omicron/blob/8d7f973204c6fbe22a04ee0136eee9f475f34a57/nexus/src/app/quiesce.rs#L362-L380

These are operational errors. They're not under the program's control and so we definitely don't want to panic. But we also expect them to never happen in a deployed system. If it did, we'd want to know and we'd want to debug it. It would be great if we had an interface for reporting these. Maybe it could emit an ereport and the fault management subsystem could turn that into an active problem? Or we could drop a little report into the RFD 613 debug dropbox (but then we'd need some way to find these and generate an active problem).

Relatedly, although we have historically advised panicking on all programmer errors (that is, non-operational errors -- things that shouldn't be possible in a working program), that guidance was oriented around situations where the blast radius cannot be well-understood at development-time: unwrap()s that the programmer thought literally couldn't fail, invariant violations (assertion failures), memory corruption, division by zero, a true null pointer dereference, etc. In practice, we've run into many programmer error-type scenarios (where the problem is fully in control of the program and really shouldn't ever be possible) where the blast radius is well-understood. Some examples from sled agent:
https://github.com/oxidecomputer/omicron/blob/8d7f973204c6fbe22a04ee0136eee9f475f34a57/sled-agent/config-reconciler/src/debug_collector/handle.rs#L167-L188
https://github.com/oxidecomputer/omicron/blob/8d7f973204c6fbe22a04ee0136eee9f475f34a57/sled-agent/config-reconciler/src/debug_collector/file_archiver/filesystem.rs#L79-L87

Particularly in Nexus, it's usually better not to panic in these cases because it's not necessary (if the blast radius is understood and bounded) and the impact of doing that in Nexus is potentially huge. (A single restart isn't a big deal, but often the state that caused it is persistent, meaning Nexus will enter a restart loop and eventually be removed from service. That's pretty bad by itself, but if another Nexus winds up taking over the work, we can take out the whole control plane this way.) But we still want to know if these things happen and we want to have the information needed to debug them.

Start by reading the cited examples in nexus/db-queries/src/db/datastore/deployment.rs, nexus/reconfigurator/planning/src/mgs_updates/rot_bootloader.rs, nexus/src/app/quiesce.rs, and the sled-agent debug collector files. Compare the possible ereport and RFD 613 debug-dropbox paths, then define an interface and reporting behavior for bounded impossible situations. Done means the design identifies how reports become discoverable active problems without panicking.