Sled agent hung on `zfs get` process accessing unhealthy U.2 drive

During a customer upgrade, we found that one of the U.2 drives on the sled was unresponsive and marked as dead:

> *nvme_state::walk softstate | ::print nvme_t n_dead
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0 (0)
n_dead = 0x1 (B_TRUE)
n_dead = 0 (0)
n_dead = 0 (0)

The sled-agent process on this sled was unresponsive to API calls and had ceased logging new events for several days. Check ptree, we found it was waiting on a zfs get child process:

642    ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxi
  643    /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml
    28339  /usr/sbin/zfs get -Hpo value mounted oxp_4e41c27f-ef7b-48f1-98be-c24f18cfd53b

This process did not respond to attempts to kill it:

BRM27230020 # kill 28339
BRM27230020 # ps aux | grep 28339
root     28339  0.0  0.0 8044 4408 ?        S   Sep 17  0:00 /usr/sbin/zfs get 
BRM27230020 # kill -9 28339
BRM27230020 # ps aux | grep 28339
root     28339  0.0  0.0 8044 4408 ?        S   Sep 17  0:00 /usr/sbin/zfs get 

Running pstack against the sled-agent process, we see the following backtrace:

------- thread# 125 / lwp# 125 [tokio-runtime-worker] --------
 fffff5ffef0dd65a pollsys  (fffff5ffcf1fe280, 2, 0, 0)
 fffff5ffef06b57a poll (fffff5ffcf1fe280, 2, ffffffff) + 5a
 000000000481ee7f _ZN3std3sys3pal4unix4pipe5read217hc9bca2dd9f0c52f2E () + df
 0000000004816621 _ZN3std7process7Command6output17h0131c7eac3b4aa8aE () + 111
 00000000026d8824 _ZN122_$LT$sled_agent_config_reconciler..dump_setup..RealZfs$u20$as$u20$sled_agent_config_reconciler..dump_setup..ZfsInvoker$GT$12zfs_get_prop17hc2ebf632a109ab15E () + a4
 00000000026d7eca _ZN28sled_agent_config_reconciler10dump_setup13GetMountpoint10mountpoint17h4e034c86d00d1fcaE () + aa
 000000000271491d _ZN106_$LT$core..iter..adapters..flatten..Flatten$LT$I$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$4next17h162d77ee11d6694eE () + bd
 000000000271c4f0 _ZN98_$LT$alloc..vec..Vec$LT$T$GT$$u20$as$u20$alloc..vec..spec_from_iter..SpecFromIter$LT$T$C$I$GT$$GT$9from_iter17h6e6a47a7a2a748a8E () + 30
 00000000026d8e7b _ZN28sled_agent_config_reconciler10dump_setup15DumpSetupWorker19update_mounted_dirs17h6f88fd58fc5c1424E () + 7b
 00000000027696fb _ZN28sled_agent_config_reconciler10dump_setup15DumpSetupWorker18reevaluate_choices28_$u7b$$u7b$closure$u7d$$u7d$17h17533ff6ca85334aE () + 5b
 0000000002766352 _ZN28sled_agent_config_reconciler10dump_setup9DumpSetup3new28_$u7b$$u7b$closure$u7d$$u7d$17h7e61630964fa761fE.llvm.9186878032128327106 () + 782
 0000000002775b8c _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17h648de51fb561900bE () + 7c
 00000000046245f3 _ZN5tokio7runtime9scheduler12multi_thread6worker7Context8run_task17ha05bf6d07bff15b6E () + 143
 0000000004623ddb _ZN5tokio7runtime9scheduler12multi_thread6worker7Context3run17h6aba92c6c6987980E () + 95b
 0000000004659baa _ZN5tokio7runtime7context6scoped15Scoped$LT$T$GT$3set17h65cd37cc73033cccE () + 2a
 000000000465c1d4 _ZN5tokio7runtime7context7runtime13enter_runtime17h4e17922ab4048628E () + 194
 00000000046233e0 _ZN5tokio7runtime9scheduler12multi_thread6worker3run17hce92c4b89c96de8eE () + a0
 0000000004629630 _ZN5tokio7runtime4task4core17Core$LT$T$C$S$GT$4poll17had8a5d1b041431a3E () + 70
 000000000461d30a _ZN5tokio7runtime4task7harness20Harness$LT$T$C$S$GT$4poll17hc16cf675f6ed30a9E () + 8a
 000000000464e524 _ZN5tokio7runtime8blocking4pool5Inner3run17h1f3459f768ebbf5eE () + e4
 0000000004657e52 _ZN3std3sys9backtrace28__rust_begin_short_backtrace17h5ebef461d95176b1E () + 72
 00000000046584b7 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17h79c1794a14872a60E () + 97
 000000000481fbcb _ZN3std3sys3pal4unix6thread6Thread3new12thread_start17h8d13b9564d02f57cE () + 2b
 fffff5ffef0d6777 _thrp_setup (fffff5ffede6da40) + 77
 fffff5ffef0d6ac0 _lwp_start ()

Which I think would correspond with:

• RealZfs::zfs_get_prop
• GetMountpoint::mountpoint
• DumpSetupWorker::update_mounted_dirs
• DumpSetupWorker::reevaluate_choices
• DumpSetupWorker::poll_file_archival

This is an run in an independent task that is launched during startup, and shouldn't impact other tasks when well-behaved. I suspect that this is causing the entire process to hang because we're using the blocking std:process API to fork zfs get, rather than tokio::process.

Start in sled-agent/config-reconciler/src/dump_setup.rs, especially RealZfs::zfs_get_prop, GetMountpoint::mountpoint, and the DumpSetupWorker methods named in the report. Trace how the zfs get child is launched and awaited, then investigate the behavior when it cannot be killed. Done means an unresponsive zfs get cannot prevent sled-agent from serving API calls or processing other tasks, with coverage for the failure case.