Address test flakes with high CPU counts

Originally reported in #oxide-control-plane by @rcgoodfellow.

Ry has a Threadripper 3990X (64-core, 128-thread) machine with 256 GiB of RAM running illumos. On this system, running cargo nextest run without arguments -- which implies -j128 for Ry -- virtually guarantees that some of the tests will flake out. Here's an example: https://gist.github.com/rcgoodfellow/7b2f9c235995093c808a778bfea3c616

The tests that flake share some common characteristics:

• they're integration tests that test a lot of components and external services, mostly Nexus and everything around it
• the tests have fixed timeouts (typically 30 seconds) while starting external services
• those timeouts are hit, presumably under some kind of contention

As a result, Ry typically runs with -j16, and he said:

fwiw: every once in a while with 16 threads i'll see a failure similar to the ones i posted in the gist, but i'd guestimate that at at less than 5% of my runs

(Ry is going to try using -j32 for a bit.)

The root issue is that like programs in general, tests are bound along three axes: CPU, memory, and I/O (usually memory and disk I/O, but also in some cases network I/O). By default, tests use -jN where N is the number of threads (note: not cores, so in Ry's case, -j128). This only bounds execution based on CPUs, without taking memory or I/O into account.

For now, there are several ways to address this:

• Increase timeouts: Right now we have timeouts of 30 seconds. We could bump them up. One of the downside to that, however, is that users may have to wait longer if the system is truly stuck.
• Decrease contention: Under high load, there might be disk or memory contention that is leading to suboptimal performance. We could spend some time investigating to see if there are low-hanging fruit which can help.
• Heavy tests: Nextest has the notion of threads-required, which let you mark some tests as being heavier than others. For example, we could mark all the Nexus integration tests as taking multiple slots of concurrency (say 2). This would have to be carefully benchmarked, however, because it may end up hurting test runs on lower-core machines that aren't contended.

In the future, it would be nice to be a bit more dynamic.

• As a first step, one could imagine something like the default number of threads being specified using an arithmetic expression. For example, something like min(num_cpus(), total_memory() / 2GiB, 32).
• These expressions could also be used in threads-required -- that also could be an expression that outputs 1 on a typical high-end 16-core machine, but 2 or even 4 on a 64-core machine.
• In the limit, one could imagine a dynamic scaling algorithm -- one that observes metrics like memory and I/O use on the host system, has historical metrics per test, and scales the number of tests up and down for that. (This is basically a scheduler, and there's plenty of complexity we can throw at this if we choose. To be clear, I'm not suggesting we actually build this -- I'm proposing a utopian solution which we can practically approximate with much less work.)

Start by reproducing the flakes with cargo nextest run at -j128 and -j16, focusing on the Nexus integration tests and their fixed startup timeouts. Review nextest's threads-required support and compare the proposed timeout, contention, and concurrency approaches; done requires an agreed mitigation with reliable high-CPU test runs.