further tune sled-agent memory budgeting

the Omicron side of my RFD 413 refresh and an extension of https://github.com/oxidecomputer/omicron/issues/7448.

https://github.com/oxidecomputer/omicron/issues/7448 was resolved with an improvement to the VMM reservoir calculation but there are further steps for improvement here. in this thread Greg and I discussed the tradeoffs of being more precise with budget terms today. we settled on not being as prescriptive as we could, today, for a few reasons:

• we'd want to start the control_plane_memory_earmark_mb setting much higher than it is: in the range of 170 GiB for current systems
• we'd want to set vmm_reservoir_percentage much higher (90+%)
• we'd really want to be able to tune the control plane earmark down once sled-agent knows which control plane services are not on its sled
• we really want to better understand long-uptime and peak usage behaviors of all services

the first two points here conflict with our relatively poor characterization of the last point.

• we can do better about measuring propolis (https://github.com/oxidecomputer/propolis/issues/890) and crucible (https://github.com/oxidecomputer/crucible/issues/1692) over time. this will give us more confidence about stability with a higher vmm_reservoir_percentage
• tune control plane earmark down from an initial conservative guess. this is only half thought-through; if we size the reservoir too aggressively and need to shrink it to move a new CRDB instance onto a sled, there's clearly some overhead there. over long uptimes, presumably this would contribute to fragmenting the VMM reservoir too.
• we should collect memory use of other control plane services to inform these calculations, but as with the Propolis and Crucible issues, collecting address space metrics currently induces stalls in the measured processes, so we wouldn't want to do this in an unattended regular manner.

some additional work to refine RFD 413 numbers:

• "Network slush buffer" describes 18 GiB for buffers handling a 100gbps network link. this is probably an overestimate (it comes from some very boring math: 100 Gbps / 8 yields 12.5GiB per second of buffered data at line rate. so 1.5 seconds gets you 18.75GiB. but in reality on the tx path buffers are loaned from the guest, so assuming jumbo frames that's roughly 9000 that live in the guest plus some structures on the host describing the intended transmission. on the rx path we presumably have to allocate on the host, but there are probably settings for the driver that control how much allocation might be done.
• before I was hoping to measure process memory use, but as @leftwo has pointed out zonestat at least accounts everything in a zone. this is more correct for "how much memory do we use in service of a Propolis" anyway (and is what Josh and Patrick were trying to impress upon me a while ago..). it'd be good to pull this into oximeter or something to see what high watermarks are like across various load tests.