understand (and possibly improve) instance creation times

In recent experience with rack2, we have had a few discussions around the time it takes to create instances with a lot of memory. Some examples are: the as of not yet understood https://github.com/oxidecomputer/omicron/issues/3417, data from @askfongjojo that indicates large instances often reliably take around 40 seconds to create and start, and observations that creation of large instances often times out.

Recently we landed support in omicron for using the VMM reservoir, which helped alleviate some of the pain around creating large instances, but it still takes on the order of 30+ seconds to create instances > 64GiB of memory, so I wanted to understand where that time was going.

I looked at a couple of larger instances this week on the dogfood cluster, and saw that there was about 20-25 seconds for a 64gb/96gb memory instance between the first propolis-server log line and the log line indicating a VNIC was being created for instance. (I intended to look at more, smaller instances, but was hamstrung by unrelated issues.) In between those two events, by code inspection I see that we would make OS call to allocate guest memory from the reservoir. @pfmooney did some testing of large VMs and found that the actual reservoir allocation was very small (order of microseconds), but it took around 15 seconds to map ~60GiB of memory into the guest address space. It thus seems plausible but that's where our time was spent, but we have little in the way of logging to show that.

It does not seem that improving instance creation times for large VMs is a big priority at the moment (though of course, no one is going to complain if instance creation is faster!). That said, from looking at this issue so far, it's clear that we could have better data here. At a minimum, I think we should:

• look at ways we can improve the propolis-server logging to understand where time is going during instance creation
• take a similar tack on the omicron side with https://github.com/oxidecomputer/omicron/issues/3877

Start with bin/propolis-server/src/lib/initializer.rs and the log line for VNIC creation, then trace the instance-creation steps between the first server log and that event. Compare timestamps for large instances and inspect the guest-memory reservoir mapping path described in the issue. Done means propolis-server logs identify where creation time is spent, with the related omicron investigation tracked separately in issue 3877.