Investigate write IO latency

When running a fio write workload from an ubuntu guest, I see some write outliers:
I have fio:

[global]
ioengine=libaio
direct=1
bs=4k
iodepth=32
rw=randwrite
numjobs=140

# This keeps us from destroying the disk label
# It also assumes we have a 5T disk
offset=1M
# Total size, 500m - 1024*1024
size=536869863424

[nvme2]
filename=/dev/disk/by-id/nvme-nvme.01de-637275746573742d6461746131--00000001

From dtrace -s /opt/oxide/crucible_utils/perf-vol.d
Updated to just look at one UUID:

  937ccd53-e8f0-4905-a8d6-e04df64a495f                write   467001

           value  ------------- Distribution ------------- count    
            4096 |                                         0        
            8192 |                                         36       
           16384 |                                         5554     
           32768 |@@@@@@@@@                                108436   
           65536 |@@@@@@@@@@@@@@@@@@@@@                    240371   
          131072 |@@@@@@                                   72052    
          262144 |@                                        10796    
          524288 |                                         186      
         1048576 |                                         10       
         2097152 |                                         0        
         4194304 |                                         0        
         8388608 |                                         64       
        16777216 |                                         152      
        33554432 |@                                        13778    
        67108864 |@                                        15566    
       134217728 |                                         0 

Some of those IOs are taking a bit longer. Worth investigations as well as what dtrace probes in propolis report.

Start by reproducing the fio write workload from the issue on an Ubuntu guest and run /opt/oxide/crucible_utils/perf-vol.d, including its UUID-focused update. Compare the observed outliers with what dtrace probes in propolis report; done means identifying where the latency is introduced and documenting the result.