If local disk is lagging behind S3, then new snapshot might overwrite the S3 snapshot

Say we have a database and the contents of the disk lag behind S3. That is, S3 has a snapshot taken at t2 and disk reflects some previous time t1 (this can happen if the volume is restored from Fly from a previous back up)

Here is how you can reproduce this:

1. t1: Create a db and insert 5 rows. Copy the sqld.data to some place
2. t2: Insert a few more rows. Total row count 10
3. t3: Shutdown sqld
4. Run bottomless-cli and restore the db, row count is 10, as expected
5. Move t1 data into sqld.data
6. t4: Restart sqld. Let it do a snapshot and upload

now on bottomless-cli, do a plain restore, it shows only 5 rows

bug:

• before snapshotting, it should check whether the current data on disk is latest than S3
• bottomless needs to restore to latest data

I think this can lead to data loss, if bottomless cleans up and deletes older-looking generations

$ bottomless-cli -n 'ns-avidb:eb4b69ea-b087-4565-ade4-a111801e6a5b' -d db ls

e4e55729-f1ee-7b13-b386-afddadbfec53 (created: 2023-10-12T08:43:10+00:00) <- t1 data
e4e55747-5808-7661-bc96-e922f40678e4 (created: 2023-10-12T08:11:03+00:00) <- t2 data
e4e55762-a1ed-794d-bc2d-26d30bf380fc (created: 2023-10-12T07:41:15+00:00) <- t1 data

Reproduce the sequence using bottomless-cli: create t1 and t2 data, restore t2, replace sqld.data with t1, then restart and snapshot. Read the snapshot and restore flow to determine how the disk generation is compared with S3. Done means a stale local disk cannot overwrite the newer remote snapshot and plain restore returns the latest data.