Consider compressing debug datasets with lz4 instead of gzip-9

The crypt/debug datasets are compressed with gzip-9 in an effort to ensure we can retain the maximum amount of logs and core dumps in the 100G quota we set per U.2 drive. gzip-9 is an extremely slow compression method, and has been running into issues with allocation when the system is under memory pressure. lz4 is available today (zstd is not available yet on illumos ZFS), so let's see how much more space is used with that enabled.

On dogfood, I created a new dataset using the same settings as crypt/debug, but with compression set to lz4, then copied the original's data over using rsync. We see a 1.46x increase in space used:

BRM13250012 # zfs get -o name,property,value encryption,compress,quota,used oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug-lz4-test                                  
NAME                                                           PROPERTY     VALUE
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug           encryption   aes-256-gcm
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug           compression  gzip-9
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug           quota        100G
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug           used         6.72G
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug-lz4-test  encryption   aes-256-gcm
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug-lz4-test  compression  lz4
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug-lz4-test  quota        100G
oxp_47b3a5b6-0f9e-4a13-97d3-fc50d37e5463/crypt/debug-lz4-test  used         9.81G

The existing data was entirely log files, so to check if core/crash dumps would have a significantly different compression ratio I took a core of sled-agent. Looks about the same, 1.42x larger on lz4.

BRM13250012 # du -k debug/test-core.666 debug-lz4-test/test-core.666                                                                                                                                                            
122673  debug/test-core.666
173629  debug-lz4-test/test-core.666

The largest debug dataset on dogfood today is at 91.0G, so switching it to lz4 would push it over its quota. sled-agent should prune it to get under 80G, but I'm not sure what would happen to log archiving in the meantime. However, most datasets are close to 20G, and would be fine.