Investigate file backend performance

File based backend performance interest has gone up in recent years.
Let's look at what the current state of things is and consider what we can do to make it go faster.

Setup steps. Create a zvol with 4k block size:

zfs create -V 500G -o volblocksize=4k oxp_247f8845-8f59-4669-a503-83e754c32327/crucible/local

Create your network device:

dladm create-vnic -t -l igb0 -m 08:00:20:ac:e9:16 vnic_prop0

Run propolis server:

./propolis-server run /alan/vm/OVMF_CODE.fd 0.0.0.0:55440

Using this config file: alpine-cli2.toml:

[block_dev.vmroot]
type = "file"
path = "/alan/vm/vmroot-backup"

[dev.block1]
driver = "pci-nvme"
block_dev = "vmroot"
pci-path = "0.4.0"

[block_dev.local]
type = "file"
block_size = 4096
workers = 32
path = "/dev/zvol/dsk/encrypted-volume/crucible/regions/local"

[dev.block2]
driver = "pci-nvme"
block_dev = "local"
pci-path = "0.5.0"

[dev.net0]
driver = "pci-virtio-viona"
vnic = "vnic_prop0"
pci-path = "0.7.0"

Run these to commands to start the VM:

./propolis-cli -s 0.0.0.0 -p 55440 new -c 16 -m 65536 --config-toml alpine-cli2.toml crud
./propolis-cli -s 0.0.0.0 -p 55440 state run

(more coming soon)