Audit crucible for holding mutex across await and other cancelation complications

Relating to the CP huddle 6/13/23:

• Async cancellation is perilous, especially (but not only) if tokio::Mutex is involved
• Two recent bugfixes to allow long-running operations to continue even if a client disconnects:
  https://github.com/oxidecomputer/omicron/pull/3140
  https://github.com/oxidecomputer/omicron/pull/3351 (not yet merged)
• Working on a dropshot level change that will allow a server-level “detach endpoint futures from client disconnects”. This would address the biggest source of async cancellation, but it’s a bit of a bandaid because it isn’t the only one (tokio::select! is another common one). It also introduces a different class of problem, but (hopefully) a less pressing/painful one than data corruption.
  https://github.com/oxidecomputer/dropshot/pull/701
• Disappointing lack of official background material available on this topic! Some docs:
  https://github.com/cbiffle/lilos/blob/main/doc/cancellation.adoc
  https://rust-lang.github.io/wg-async/vision/roadmap/scopes.html#cancellation

We should audit crucible to see how much of this we have, and what to do when we find it.

Start by auditing the Crucible Rust code for tokio::Mutex usage across await points and cancellation through tokio::select!, using the linked PRs and cancellation references as background. Done means the audit identifies affected code and establishes what remediation is needed, but the issue does not name files or tests to begin with.