Dig into potential issue with tokio start_paused

In https://github.com/oxidecomputer/omicron/pull/2654 we found that a test with tokio::test(start_paused = true) that consistently worked on Linux was flaky on illumos.

Note that start_paused = true runs tokio in a mode where timers are paused, and auto-advance. Since all this code uses Tokio, pausing and auto-advancing timers should work in theory -- there's no reason for any of these operations to be tied to system timers.

This could be any of the following:

• an omicron code difference between Linux and illumos (though the test doesn't use any illumos-specific features and is completely OS-independent -- it uses MGS talking to sp-sim).
• a scheduler difference on illumos compared to Linux
• a Tokio difference between illumos on Linux
• a bug in Tokio

Start with the flaky test described in omicron PR #2654 and reproduce it on Linux and illumos. Compare the behavior of Tokio's start_paused = true mode and the OS scheduler, while checking for omicron-specific differences. Done means identifying the source of the discrepancy and documenting or fixing the underlying issue.