Multi-file torrents for snapshots

There are large numbers of files that are synchronized from BitTorrent as groups of related files. Those files are each in individual torrents. For p2p purposes, this requires BitTorrent clients to announce and synchronize on each individual swarm, even though all the participants are the same. This overhead appears to be between 600-1400x based on the number of torrents in a given related group of snapshot files. Announce events compete for access to tracker resources in the client, and multiple connections are made to the same peer for different torrents. Collapsing all these files into a single torrent removes this overhead, as well as simplifies operations overhead. For example, using external seedboxes or third-party torrent clients to contribute to the Erigon BitTorrent swarms becomes a case of adding a few specific torrent files.

To my understanding, new snapshot files are produced at regular intervals. The file contents are deterministic, meaning any Erigon node will agree on what those snapshot files look like, and when they should come into existence. This allows for distributed, deterministic generation of a torrent containing all relevant files. Following from that it means you have a deterministic infohash, and so a deterministic way to migrate your snapshot torrent swarm over time. Importantly, this is reliable in a distributed way: If your central GitHub/R2 or other bootstrapping facilities are unavailable, it’s still possible for the network to sync and share new snapshots. It would also be possible to initialize a new Erigon client from a single infohash.

It is possible to generate a multi-file snapshot torrent from existing snapshot files, and it's even possible to generate it from the existing single-file torrents without having to rehash all the data. You would likely use this concept when evolving your torrent to efficiently include new snapshot files. BEPs involved for this include:

- https://bittorrent.org/beps/bep_0003.html
- https://bittorrent.org/beps/bep_0047.html
- https://bittorrent.org/beps/bep_0052.html

Specifically for the need to ensure file ordering is consistent. Files are padded for v1 to ensure hashes can be rolled forward into new infos, and v2 to ensure things are done in a way compatible with v2 hashing and the piece layers.

Switch to multi-file torrents. This consolidates all data on a single swarm. In particular this makes using the DHT much more feasible. This requires clarifying how frequently snapshots are added to different chains, as if it occurs more often than about every 15-20 mins it might require some rethinking. Less often than every 2-4 hours would be very comfortable.

Advantages:

- Single BitTorrent swarm per chain. One DHT location, one tracker announce. Shared peers for all related files. This means less connection overhead, less UDP packets. Less traffic on VPS.
- Much more manageable for testing, seedboxes and upgrading config though metainfo and magnet link.

Notes:

- Requires either rehashing, or a small bit of code to generate new torrent without rehashing old data. (Provide a small tool to help). Probably will also require the use of a padding file for BitTorrent v1 (very common).
- Benefits from BitTorrent v2 but not required.

Snapshot file versioning:

If clients are generating different snapshot file versions (the format changed etc.), they would keep trying to diverge into different file sets. I think the easy solution is to allow them to do this, it would mean different client versions would maintain different swarms if their preferred snapshot versions differed.

Sub issues:

- Changes to snapshot generation.