Local storage on write-cacheful disks

in https://github.com/oxidecomputer/omicron/pull/10932, the local volume semantics are "if the disk is backed by /dev/zvol/rdsk/, it is assumed to be on the U.2s we test in the product, which to date and for the foreseeable future are not expected to report volatile write caches".

this is wrong in two important ways:

• it is technically possible that a U.2 in the field will spontaneously grow a write cache (this what NVMe Get Log Page - SMART Health Information Log bit 4 is about: volatile memory is present and the memory backup solution failed)
• development environments (like my workstation!) probably have commodity storage devices that very much have volatile write caches.

#10932 elaborates on the costs of "wrong" here, but suffice to say we want to not be wrong.

in the U.2 case, there are many questions. do we place new local volumes on partially-degraded U.2s? should we consider this kind of degradation tolerable? does a local volume on such a U.device have the "faulted" state, or can new instances with existing volumes here start (say, to evacuate data)?

for the cases of disparate hardware, we'd really need sled-agent to query disks for write-cachefulness and consider that in the control plane. we're also assuming that, for example, no one is going to randomly change the enabledness of a disk's write cache.

for the "degraded-in-the-field" case, regardless of the InstanceSpec Nexus sends along, Propolis needs to be aware of the actual hardware state to tell guests if the storage device underlying a local volume has experienced a failure. that's a distinct issue, Propolis#1185. this may come with a device reporting a VWC when it did not before - how disks degrade here is (to me!) kind of a string of unknowns.

Start with PR #10932, which provides the context for the current local-volume semantics. Then trace how sled-agent, the control plane, Nexus, and InstanceSpec represent disk write-cache state, while reviewing Propolis#1185 for the guest-visible failure behavior. Done requires an agreed policy for degraded U.2 devices and disparate hardware before implementation scope can be defined.