Inventory-based backup tool

A solution (or part of it) for
- dandi/dandi-utils#2

We have inventory which gives us transactional log of what has happened to the bucket. We have it dumped also into the bucket:

```
dandi@drogon:/tmp$ aws s3 ls s3://dandiarchive/dandiarchive/dandiarchive/ | headtail
PRE 2019-10-02T04-00Z/
PRE 2019-10-03T04-00Z/
PRE 2019-10-04T04-00Z/
....
PRE 2024-11-12T01-00Z/
PRE 2024-11-13T01-00Z/
PRE data/
PRE hive/
```

(note - I do not think that beginning date is the ultimate beginning of the bucket unfortunately but probably ok), where for each dated folder we have

```
dandi@drogon:/tmp$ aws s3 ls s3://dandiarchive/dandiarchive/dandiarchive/2024-11-06T01-00Z/
2024-11-06 14:31:05 33 manifest.checksum
2024-11-06 14:31:05 72797 manifest.json

dandi@drogon:/tmp$ aws s3 cp s3://dandiarchive/dandiarchive/dandiarchive/2024-11-06T01-00Z/manifest.checksum .
download: s3://dandiarchive/dandiarchive/dandiarchive/2024-11-06T01-00Z/manifest.checksum to ./manifest.checksum
dandi@drogon:/tmp$ aws s3 cp s3://dandiarchive/dandiarchive/dandiarchive/2024-11-06T01-00Z/manifest.json .
download: s3://dandiarchive/dandiarchive/dandiarchive/2024-11-06T01-00Z/manifest.json to ./manifest.json
dandi@drogon:/tmp$ cat manifest.checksum
03260719490f9dc0665b9d281fd4180a
dandi@drogon:/tmp$ md5sum manifest.json
03260719490f9dc0665b9d281fd4180a manifest.json
```

and the manifest.json is actually pointers to the listing of items in the bucket

```yaml
# dandi@drogon:/tmp$ head -n 15 manifest.json
{
"sourceBucket" : "dandiarchive",
"destinationBucket" : "arn:aws:s3:::dandiarchive",
"version" : "2016-11-30",
"creationTimestamp" : "1730854800000",
"fileFormat" : "CSV",
"fileSchema" : "Bucket, Key, VersionId, IsLatest, IsDeleteMarker, Size, LastModifiedDate, ETag, IsMultipartUploaded",
"files" : [ {
"key" : "dandiarchive/dandiarchive/data/15a3a67a-6dec-44b6-9800-f8ddf5a44870.csv.gz",
"size" : 169006021,
"MD5checksum" : "bf6060d5dbf25c6443712b9846e812c2"
}, {
"key" : "dandiarchive/dandiarchive/data/0f25f9ec-cfc4-4e5a-932c-9a5ca5c2c8aa.csv.gz",
"size" : 49364618,
"MD5checksum" : "6d57525d105ef36e8228a408e4de29cc"
```

and those .csv is the compressed listings

```shell
dandi@drogon:/tmp$ aws s3 cp s3://dandiarchive/dandiarchive/dandiarchive/data/15a3a67a-6dec-44b6-9800-f8ddf5a44870.csv.gz .
download: s3://dandiarchive/dandiarchive/dandiarchive/data/15a3a67a-6dec-44b6-9800-f8ddf5a44870.csv.gz to ./15a3a67a-6dec-44b6-9800-f8ddf5a44870.csv.gz
```

```
dandi@drogon:/tmp$ zcat 15a3a67a-6dec-44b6-9800-f8ddf5a44870.csv.gz | head -n 3
"dandiarchive","zarr/73107a2a-9eb2-47ed-be62-1feef97b8026/1/0/0/5/5/76","J_lZKtM.5fliCCeIGTmufk3R9lc_FmZg","true","false","904","2022-04-09T21:52:14.000Z","fbebe2d44529d0a25247e8b5bca5956b","false"
"dandiarchive","zarr/73107a2a-9eb2-47ed-be62-1feef97b8026/1/0/0/5/5/77","xy7HOh0ZZKXczghVqPCdijvxKQMAG_04","true","false","1548","2022-04-09T21:52:02.000Z","e0c48b2230b51636e3df32f3a15b65b8","false"
"dandiarchive","zarr/73107a2a-9eb2-47ed-be62-1feef97b8026/1/0/0/5/5/78","aVAyFLM0rDCMyaIlIWmn9j3PgANjUPuc","true","false","1578","2022-04-09T21:52:02.000Z","e451a3b1afd3cff713a2a1d7120d68ba","false"
```

We need a tool which would efficiently download and then incrementally update local backup of the bucket based on those logs. Some features to target/keep in mind:

- should be efficient, so process multiple paths (keys) at once. So should be smart to not process two transactions for the same key in parallel etc
- ideally we should make backup immediately usable as a "mirror" of the bucket at current point in time, but also be a true backup going back, so if something was deleted -- we still have it on local drive

one possible "approach" could be
- reflect mtime for the key from inventory in the filename mtime when generating
- if key is the latest version -- store under original `path`
- store etags and versionids for keys in a folder under `.versions.json` or alike in that directory
- when key is deleted or to be replaced with another one:
- read record from `.versions.json` for `{etag}` and `{versionid}` and `mv` that `{path}` to `{path}.old.{versionid}.{etag}` (should preserve original mtime), and remove entry from `.versions.json`

this way we
- have immediate access to all the keys as on the bucket
- could always get to desired versionId of any key (either via `.versions.json` or looking through `*.old.*` files for the path
- do have `mtime` (stored within filesystem) so we could prune some "timed out" `*.old.* ` with a simple `find` command
- avoid using symlinks etc - only 1 extra inode for that `.versions.json` in each folder though .

There is also that `hive/`:

```
dandi@drogon:/tmp$ aws s3 ls s3://dandiarchive/dandiarchive/dandiarchive/hive/ | head
PRE dt=2019-10-02-04-00/
PRE dt=2019-10-03-04-00/
PRE dt=2019-10-04-04-00/
PRE dt=2019-10-05-04-00/
...
dandi@drogon:/tmp$ aws s3 ls s3://dandiarchive/dandiarchive/dandiarchive/hive/dt=2019-10-02-04-00/
2019-10-02 18:36:10 92 symlink.txt
dandi@drogon:/tmp$ aws s3 cp s3://dandiarchive/dandiarchive/dandiarchive/hive/dt=2019-10-02-04-00/symlink.txt .
download: s3://dandiarchive/dandiarchive/dandiarchive/hive/dt=2019-10-02-04-00/symlink.txt to ./symlink.txt
dandi@drogon:/tmp$ cat symlink.txt
s3://dandiarchive/dandiarchive/dandiarchive/data/d8dd3e2b-8f74-494b-9370-9e3a6c69e2b0.csv.gz

dandi@drogon:/tmp$ zcat ./d8dd3e2b-8f74-494b-9370-9e3a6c69e2b0.csv.gz
"dandiarchive","dandiarchive/dandiarchive/data/","XhLH9OkJH9aPWihWcbMKDzL5JGtBxgkt","true","false","0","2019-10-02T00:08:02.000Z","d41d8cd98f00b204e9800998ecf8427e","false"
```

which I do not know yet what it is about.

NB ATM I am running a sync of the inventory under `drogon:/mnt/backup/dandi/dandiarchive-inventory`