borg2: work in progress...

update: as there was no negative feedback from alpha testing, borg2 branch was merged into master, thus that big change in form of a major / breaking borg 2.0 release is coming.

read below about what's planned and what's already done.

----

what could be done if we decide to make a breaking release (2.0) that:
- does not try to be compatible with old repos
- does not try to be 100% compatible with old cli syntax (but 90%)
- only uses new repos / keys it created itself
- only gets old archives via borg import-tar or borg transfer

putting all the breaking stuff into 1 release is good for users (1 time effort), but will take quite some time to test and release.

After borg 2.0, we'll make a N+1 release (2.1? 3.0?) that drops all the legacy stuff from the codebase, including the converter for borg < 2.0 repos.

## borg 2.0 general comments

DONE: offer a `borg transfer` command, #6663, that transforms old stuff only to stuff that will still be supported by borg N+1.

## N+1 general comments

much of the stuff described here has own tickets, see "breaking" label / add issue links here.

## 2.0 crypto

- DONE repo-create: do not create old keys (pbkdf2, legacy AES class, encrypt-and-mac)
- DONE repo-create: do not create AES-CTR based repos, only new AEAD ciphers with session keys
- DONE: remove all docs talking about potential nonce reuse, counter management and related
- DONE: remove key algorithm change (pbkdf2<->argon2), just use argon2 for new repos/key
- DONE: nonce management code for aes-ctr, not needed any more with session keys, remove nonces module #7556
- keep old crypto code, we need it to read / decrypt old repos

## N+1 crypto

- remove pbkdf2 + pbkdfs/sha256 keys + docs - we have argon2 now
- remove low_level.AES class which is only used for pbkdf2 key encryption
- remove aes-ctr mode
- remove support for super-legacy passphrase-key type (not supported since long)
- we used hmac-sha256 and blake2b as id-hashes in the past, thus we need to keep them because we need an efficient `borg transfer` (not needing to re-hash)

## 2.0 keys
- DONE it is now possible to have multiple keys per repository. this can be used for better redundancy, to use different passphrases. all keys contain the same secret key material.
- DONE decouple key location from encryption type byte in repository. borg will just try to unlock both matching keyfiles and repokeys with a given passphrase until it succeeds.
- DONE keys are stored using their sha256 content-hash as name. accidental corruption can be easily detected.

## 2.0 repo

- DONE: implement new repository based on `borgstore`, #8332
- DONE: implement sftp: borgstore backend (a remote backend that does not need borg serve on the remote).
- DONE keep support for reading borg 1.x repos
- DONE read-compatibility with old local repos for `borg transfer`, and/or
- DONE read-compatibility with old RPC (ssh: repos) for `borg transfer` (in that case the old repo would be served by an old borg version)
- DONE borg check only checks new repos, no support for old repos
- DONE only generate latest hints format (this is done since long)
- DONE remove detection of attic repos, #6859
- DONE remove free nonce / nonce reservation api

## N+1 repo

- remove support for reading borg 1.x repos
- if we reduce MAX_OBJECT_SIZE from ~20MiB to 16.000.000, 24bit are enough for the entry length
- better alignment for segment entries together with the 1 type byte.
- in-memory indexes have 8 free bits without using more memory
- max archive size goes down by 20%!
- do not allow 16MiB objects, we need some room for potential header size increase in future
- **nope, we cannot do that:** we need an efficient `borg transfer`, not needing to re-chunk content!

## 2.0 indexes / cache

- DONE remove `legacy_cleanup` function
- DONE chunks index stored in repo

## 2.0 msgpack

- DONE keep bigint for reading old stuff, write with Timestamp instead of bigint #2323
- DONE read legacy str/bytes types for reading old stuff, write only more modern msgpack types (for str vs bytes), #968

## N+1 msgpack

- drop support for bigint stuff #2323
- drop support for legacy msgpack str/bytes types #968

## 2.0 archive / item

- DONE always have borg12_meta in newly created archives (includes `borg transfer`) (note: `Archive.save()` adds that)
- DONE #6763 we always have the borg12_meta, we don't have "old" archives: remove support cache for old archives
- DONE #6763 drop support for csize in item chunk list #2357
- DONE transfer: clean stoneage attic bug leftovers, remove erroneous b'acl' key from item metadata
- DONE remove stoneage attic bug support in borg check -> valid_item (erroneous b'acl' key)
- DONE upgrade items with chunks to always have precomputed `size`
- DONE #6763 read: also still support csize in item chunk list #2357 ("support" here means to tolerate 3-tuples and just throw the csize away and make a 2-tuple of it)
- DONE #6763 write: get rid of csize in item chunk list #2357 (borg transfer)
- get rid of hardlink_master and complex code dealing with it, tends to cause issues. #855 #2325
- DONE: borg transfer converts this
- DONE: borg create / extract needs to work the same way with hlid
- DONE: borg recreate
- "DONE": borg import-tar - **drop support for hardlinks for now**, see ticket ...
- DONE: borg export-tar
- DONE: borg import/export-tar BORG mode
- DONE #2388
- DONE #5607
- resolve dual use of item.source (hardlink and symlink), #2343
- DONE: borg transfer converts this
- DONE: borg create / extract needs to work the same way with hlid
- DONE: borg recreate
- DONE: borg import/export-tar BORG mode
- DONE: remove chunks_healthy and handle missing chunks dynamically, #8559
- DONE rename item.source to item.target for symlinks?
- rename item.bsdflags to item.flags or item.fileflags?

## N+1 archive / item

- Item.get_size: remove support for items with chunks, but without precomputed `size`

## 2.0 or N+1 checksums

- DONE we could even consider removing libdeflate in 2.0. the only major user will be "borg transfer" and that will be a one-time per repo usage.
- DONE remove libdeflate again and use zlib.crc32 from stdlib, PUT2 format only uses crc32 for header data, not much data getting crc'ed

## 2.0 compression

- DONE borg transfer can remove the dirty type bytes hack for zlib, add cleaner dispatch / new handler
- DONE we need to keep all compression algorithms, so borg transfer does not need to recompress
- DONE #6701
- DONE #6698

## N+1 compression

- drop support (dispatching / handler) for the zlib dirty type bytes hack (ZLIB_legacy)
- we need to keep all other compression algorithms, because borg transfer did not recompress

## 2.0 upgrade

- DONE remove borg upgrade, doing upgrades from attic / old borg (they need to first upgrade to 1.2 and then use borg transfer)

## N+1 archiver

- remove unneeded stuff from benchmark cpu

## 2.0 remote
- DONE dropped support for "ssh:" remote repositories, except for "borg transfer" from borg 1.x repositories.
- DONE implement remote repositories using "rest:" (implemented in `borgstore` 0.5.1+). borg.repository and the REST backend of borgstore is running on the client, talking REST http over stdio over ssh to a borgstore REST server on the server.

## N+1 remote
- remove legacy "ssh:" support

## 2.0 cli

- DONE separate archive name from repo url (our regexes are way too complex), #948
- DONE drop scp syntax for the repo location #6691
- DONE #6269
- DONE #6756
- DONE #6551 (jsonargparse, config, env vars)

## 2.0 locking

- DONE implement borgstore based locking for borgstore based repos.
- DONE stale lock removal of old locks that did not get refreshed.
- DONE stale lock removal of locks of dead processes.
- DONE most commands now use a shared lock, except borg compact and borg check.

## y2038 and requiring 64bit
- DONE - set SUPPORT_32BIT_PLATFORMS = False, for details see #9430.

## 2.0 speedups
- blake3 (multithreaded for big chunks)
- zstd (multithreaded for big chunks)
- new and tuned chunkers: fastcdc and *-aes using AVX2, AVX-512, NEON, VAES, AES-NI
- background store-thread (stores away a pack while borg assembles the next one)
- full-file digests are computed in a separate thread

# stuff that is out of scope

as you see above, there is already a huge scope of what should be done.

to not grow the scope even further, some stuff shall not be done (now):
- no public key cryptography (neither by gpg nor by reinventing gpg for borg)
- no full multithreading in borg (like having borg workers communicating via queues, etc.)