Closed DHT Based Routing

Libp2p currently relies on a fully p2p Kademlia DHT. Unfortunately, even if we get it to the point where it behaves optimally, it still won't scale enough to support our content routing needs.

Proposal: Implement a "closed" DHT with known members. Note: This is often called a "Distributed KV Store".

Unlike our Kademlia DHT:

1. This system would be permissioned instead of permissionless. Joining as a DHT "server" would require either human or blockchain consensus.
2. Have a fixed, well-known routing table, fetched when initially connecting to this DHT.

Motivations:

• 1-RTT lookup
• batch put
• long-lived routing records

Performing a 1-RTT lookup and/or a batch put requires a known routing table. The round-trips in traditional p2p DHTs all come from discovering this routing table along the way.

A known routing table requires some form of consensus on the members of this routing table. That's where the trust comes in.

The last part of this is long-lived routing records. The Internet Archive has ~400e9 files which equates to at least 35TiB of "provider records". However, the IA isn't adding 400e9 files per day. Given stable nodes that can be trusted to keep long-lived records alive, the IA wouldn't have to keep re-broadcasting old records to keep them alive.

Notes:

• Really, we may want to treat large services like the IA as "trackers". However, the current system won't even scale for smaller services.
• We may also want to integrate payment for large users to prevent abuse but we can probably punt on that for now.

CC @Kubuxu, IIRC you already proposed something kind of like this but I couldn't find the proposal.

CC @obo20 as this is really important for pinning services.

No files, tests, or entry points are named. Start by reviewing the proposal and related discussion to determine the membership, routing-table, consensus, and record-lifetime requirements. Done would require an agreed design and an identified implementation plan.