Further developpment

Hello Michael,

I've been running vmsync on a couple of systems, and found a couple of blocker issues.
I also understand that you have other projects to gather, so I decided to have a look myself.

As I told you, I'm not a go developper myself, but understand quite well the stakes of correctness regarding disaster recovery and general architecture.
I (or IA and I) made a couple of fixes in order to smoothen out usage.

After getting a bit more intimate with the codebase, I decided to see whether it was possible to implement zstd compression and buffering, and the answer is YES !

After having added those, I also found out that using async Pread/Pwrite with NBD and a IO queue depth highly improved overall performance, so I decided to implement it.

I ran some synthetic benchmarks around full syncs of an actual production VM.
I actually ran a lot of VM sync worloads, but this one I documented properly.
I ran on an enterprise grade WAN link which I voluntary capped to 600Mbit/s via tc to avoid having 95th percentile variances.
Transfer was done via a wireguard tunnel, with median latency of ~ 25ms.

| Version| Transfer size | NBD calculated transfer speed| Async | Compression | Compress Ratio | Buffering | IO Depth | Wall Time |
|---------|---------------|----------------------------------|--------|----------------|-------------------|-----------|---------|-----------|
| vmsync 0.27 | 8,007GB | 34.53 MiB/s | | | | | 1 | 4m0,522s |
| vmsync 0.30 | 8,007GB | 56.66 MiB/s | X | | | | 8 | 2m31,054s|
| vmsync 0.30 | 8,007GB | 83.53 MiB/s | X | zstd-3 | 75.8% | 1M,256M | 8 | 1m46,425s |
| vmsync 0.30 | 8,007GB | 101.42 MiB/s | X | s2-default| 69.7% | 1M,256M | 8 | 1m28,506s |
| vmsync 0.30 | 8,007GB | 100.22 MiB/s | X | s2-better | 70.8% | 1M,256M | 8 | 1m29,055s |
| vmsync 0.30 | 8,007GB | 110.20 MiB/s | X | s2-better | 70.8% | 1M,256M | 32 | 1m22,542s |
| vmsync 0.30 | 8,007GB | 25.68 MiB/s | X | s2-best | 71.6% | 1M,256M | 8 | 5m26,712s |

I think there is some sweetspot with s2-default and iodepth 8.

I also thought that since we already have a SSH control channel, we might as well add an option to pass traffic thru the channel via port forwarding in order to get native encryption (spoiler: speed is as almost as good as with wireguard VPN).

In order to validate all those changes in practical checks, I've decided to implement a `-verify` option which actually compares both qcow2 files over the network. The latter one is supposed to be invoked manually or in maintenance windows, is slow and suspends the source VM, but at least I can get a guarantee that there is no data consistency problem.

I also wrote a bash script that can list all VMs, and run parallel instances of vmsync since a single nbd transfer doesn't seem to be able to exhaust my WAN links.

Once these optimizations were implemented, I decided to add metrics so I can document and have better history.
I created a Grafana dashboard on my monitoring system that makes it quicky clear what works and what doesn't:

So the point is:

I initially wanted to make small purposeful PRs so it's quite easy to review the code, but at the end of the day I created a branch with multiple features and fixes because it's really hard to make all tests per commit per branch as a one man show, and some features / fixes were dependant on each other.
I quite frankly hate myself for making a single "big branch with all(TM)", but here I am.

I was able to iron out a couple of bugs of which a major data loss when external snapshots were taken and a sync occured while this happened.
Anyway, I am still trying various corner cases and letting vmsync run for days and then run consistency checks, but so far it looks good.
I also try to interrupt incremental syncs via CTRL+C, various network outages, packet loss, and then restart a new incremental sync and have a verify pass validate data consistency.

Would you allow me to make a big PR as long as I did all tests I can imagine ?

Best regards,
Ozy.