Propolis could self-assess VM exit health

https://github.com/oxidecomputer/propolis/issues/1008, more recently https://github.com/oxidecomputer/propolis/issues/1153, https://github.com/oxidecomputer/propolis/issues/376, my own interest in disk/nic latencies, and possibly other issues have at least one shared frustration: "what was Propolis doing, instead of running the vCPU?"

In 1008, it would have been somewhat helpful to tell that vCPUs had exited in trying to do accesses to NVMe registers, and ideally which registers the CPUs were accessing. it wouldn't have led to an immediate answer, but it probably would have shortened the confusion by a day or three!

In 1153, I don't know if we can tell what happened just from the observation that a CPU was stuck. from Propolis' perspective, the fact that a vCPU was stuck for any amount of time the guest would find remarkable is a bug in its own right.

In 376, was the vCPU stuck because migration left a deice in a strange state, and an exit actually took 15s+? I think so, given that this was a repeat event at different times and not correlated with migrations themselves. What was Propolis doing, and where did we get stuck?

A few months ago I'd sketched out how I think we can diagnose this stuff, but it needs a bit of work all over the place... to try itemizing:

• propolis-server and propolis-standalone could report (or record!) "slow" VM exits
• would like to look up devices from registered addresses or I/O ports
• would like to look up device registers from address or I/O port offsets
• would like a "flight recorder" of events in Propolis (as one tendril related to #335)

the first part is one I had sketched out earlier: emitting a warn!() if an exit took more than 20ms in Propolis is relatively straightforward, but all we know in general is (in the case of MMIO or port I/O) the address, size, and direction of an access. from here we can look up .... the callback to interpret that access.

this is just a function pointer, so it takes an annoying amount of work to reverse engineer even which device was accessed. then in that device it's a similar story to work out which register was being accessed - that's typically another function that branches on an offset to some arm in a match for the accessed register, which may or may not be inlined.

I'm roughly imagining that we could add to ASpace a reference to the registered device, so we can look up an id (like a device ID, or name, identifier attached by the consumer of propolis-lib (like a SpecKey?)), and then on that device look up what would handle an I/O to a particular offset. even the name of a register would be great here!

from that point we'd need to store the observation that accessing FooCfgReg on block-bar-backend took more than a trivial amount of time. here I'm imagining that we'd have some in-memory ring buffer-like-thingy that we can dump events into, which could also record remarkable states from the guest or device emulation. did a device queue get full? did we stop processing I/Os for some amount of time? did we start an I/O which has been in-flight more than 20s, after which FooCfgReg on block-bar-backend caused vCPU 4 to stall for 22 seconds?

my hope is that we could dump this out of Propolis and get a pretty straightforward answer to at least the next place to look for issues like #1153. maybe even get it into ireports and support bundles!

Start with the propolis-server and propolis-standalone handling of VM exits, then trace ASpace device registration and the warn!() path described in the issue. The proposed work spans slow-exit reporting, device and register identification, and a flight recorder or ring buffer. Done would provide useful exit observations that can be dumped from Propolis or included in ireports and support bundles.