upstairs perf: too many job-scans in gone_too_long

fair bit of time in total_live_work at low IO request sizes with my crudd tests. 15% of the flamegraph at 256-block requests, and like 40% at 16-block requests.

thats getting called in gone_too_long after every process_new_io. the point of gone_too_long is to see if there's too many outstanding IOs (1000) to mark the downstairs as being problematic. and i wonder if we could just like. call it a little less frequently.

https://github.com/oxidecomputer/crucible/blob/6bd7f6e2a53e2558827d82b88fe5ecee3102d880/upstairs/src/lib.rs#L10017-L10023

flamegraphs, if you want to see them for yourself:

2023-05-17-crudd-full-flamegraph.zip

these are a couple weeks old at this point but were run on perfwork before it was merged, iirc.

anyways, we could maybe try to re-invent total_live_work to not be linearly traversing the set of jobs via more accounting variables. or maybe we could just call it less often. i dont think we need to call it after every single IO request comes in from the guest.

Something like running it when an IO request comes in, but only if its been EITHER <some period of time> since the last time we ran it OR <some amount of issued requests since we last ran it> might be a bit nicer, while still catching a wedged downstairs early enough that it doesn't become a problem.

Start in upstairs/src/lib.rs around lines 10017-10023 and trace process_new_io, gone_too_long, and total_live_work. Review the linked flamegraph and determine whether reducing call frequency or changing the accounting approach preserves the check for too many outstanding I/Os while reducing job scans.