test failed in CI: test_import_blocks_with_bulk_write_with_snapshot

This test failed on a CI run on #7122:
https://github.com/oxidecomputer/omicron/pull/7122/checks?check_run_id=33292942202

Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01JD5YPWVSAJWZ638EACGNQSY7/YYdhS7aq7KBkQQafhJSsfkeaQFSGzpdsO6k0G4Ikyq8RHFH3/01JD5YQTA7B9110R85N97V87PF

Excerpt from the log showing the failure:

test integration_tests::pantry::test_import_blocks_with_bulk_write_with_snapshot ... FAILED
failures:
failures:
    integration_tests::pantry::test_import_blocks_with_bulk_write_with_snapshot
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 398 filtered out; finished in 157.07s
--- STDERR:              omicron-nexus::test_all integration_tests::pantry::test_import_blocks_with_bulk_write_with_snapshot ---
log file: /var/tmp/omicron_tmp/test_all-56b0a9ae47a317de-test_import_blocks_with_bulk_write_with_snapshot.164209.0.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-56b0a9ae47a317de-test_import_blocks_with_bulk_write_with_snapshot.164209.0.log"
DB URL: postgresql://root@[::1]:37197/omicron?sslmode=disable
DB address: [::1]:37197
log file: /var/tmp/omicron_tmp/test_all-56b0a9ae47a317de-test_import_blocks_with_bulk_write_with_snapshot.164209.2.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-56b0a9ae47a317de-test_import_blocks_with_bulk_write_with_snapshot.164209.2.log"
log file: /var/tmp/omicron_tmp/test_all-56b0a9ae47a317de-test_import_blocks_with_bulk_write_with_snapshot.164209.3.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-56b0a9ae47a317de-test_import_blocks_with_bulk_write_with_snapshot.164209.3.log"
thread 'integration_tests::pantry::test_import_blocks_with_bulk_write_with_snapshot' panicked at nexus/tests/integration_tests/instances.rs:6359:19:
instance 62e7f377-c452-471d-8952-adb75e834e1e did not transition to Stopped after 120s: timed out after 120.474485563s
stack backtrace:
   0: rust_begin_unwind
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14
   2: {async_fn#0}
             at ./tests/integration_tests/instances.rs:6359:19
   3: {async_fn#0}
             at ./tests/integration_tests/instances.rs:6310:6
   4: {async_fn#0}
             at ./tests/integration_tests/pantry.rs:162:75
   5: {async_fn#0}
             at ./tests/integration_tests/pantry.rs:540:74
   6: {async_block#0}
             at ./tests/integration_tests/pantry.rs:492:1
   7: poll<&mut dyn core::future::future::Future<Output=()>>
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs:123:9
   8: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs:123:9
   9: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:696:57
  10: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:696:25
  13: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:423:19
  14: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:695:36
  15: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:774:68
  16: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9
  17: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:26
  18: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:283:12
  19: std::thread::local::LocalKey<T>::with
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:260:9
  20: tokio::runtime::context::set_scheduler
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:9
  21: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:774:27
  22: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:683:19
  23: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:191:28
  24: tokio::runtime::context::runtime::enter_runtime
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:179:9
  26: tokio::runtime::runtime::Runtime::block_on_inner
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:361:47
  27: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:335:13
  28: test_import_blocks_with_bulk_write_with_snapshot
             at ./tests/integration_tests/pantry.rs:492:1
  29: test_all::integration_tests::pantry::test_import_blocks_with_bulk_write_with_snapshot::{{closure}}
             at ./tests/integration_tests/pantry.rs:495:2
  30: core::ops::function::FnOnce::call_once
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5
  31: core::ops::function::FnOnce::call_once
             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpKBkQfp"
	If you would like to access the database for debugging, run the following:
	# Run the database
	cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpKBkQfp/data"
	# Access the database. Note the port may change if you run multiple databases.
	cockroach sql --host=localhost:32221 --insecure
WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 164250) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-56b0a9ae47a317de-test_import_blocks_with_bulk_write_with_snapshot.164209.1-clickhouse-smvbD7)
WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpVKBtFg
WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmp1deA1B
WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmprYYoyO
WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmp3tGKEW

@jmpesp can you take a look?

Start with the failing test in nexus/tests/integration_tests/pantry.rs and the timeout at nexus/tests/integration_tests/instances.rs:6359. Review the linked CI run and its log, including the leaked CockroachDB, ClickHouse, Dendrite, and Mgd processes. Done means identifying and fixing the cause so test_import_blocks_with_bulk_write_with_snapshot reaches Stopped without timing out.