RFC: magic-cookie-discovery: a zero-config, DHT-based discovery mechanism for libp2p

The following is the result of an idea I've iterated on with @mxinden and @dhuseby after IPFSthing. I am not aware of an immediate user that is asking for this but I think it might have the potential to be a "wow"-effect for libp2p for new users and I wanted to gather some feedback from the wider community. Coming from a user-perspective myself and only now being a maintainer, I would have wanted this to exist 3 years ago.

This idea is very much early stage and I am curious what other people think.

Description

Our DHT spec allows for a concept of "provider records". They are basically sets where nodes can add and remove themselves to/from a certain key. The gist of the idea is that we specify a function that deterministically computes an identifier from a node's supported protocols and we automatically add ourselves as a "provider" for this identifier. Thus, a node that supports the same set of protocols (or a subset based on configuration) can connect to an existing libp2p-DHT network (like IPFS) and find nodes that support the same set simply by looking up providers for its "identifier".

Example:

• Assume a libp2p network with a DHT
• Node A boots and only knows about a bootstrap node of this network (like an IPFS node)
• Node A looks at its protocols and deterministically computes an identifier X from it
• Node A adds itself as a provider for X

Note that being a "provider for X" doesn't have any further meaning beyond supporting the protocols being used to compute X. It doesn't mean we support BitSwap or any other protocol to actually fetch a value for X because there is no value for X.

• Node B with the same set of protocols boots and connects to the same libp2p DHT
• Node B computes the same identifier and lists all providers for X
• Node B discovers node A through this mechanism

Details

A few details on how we could implement this:

• Allow nodes to filter, which protocols contribute to the identifier, like filtering out irrelevant ones (i.e. filter out the actual DHT protocols, identify, ping etc)
• Sort remaining protocols and hash them
• Use a TOTP-based approach where the final identifier is time-based, meaning who stores the providers in the network moves around with time, rotating who in the network has to put up with the "burden" of storing these provider records
• Make sure these nodes are still good citizens of the DHT-network and not parasites

Alternatives

We could let the user decide the identifier by which they want to discover their network. This is easily possible but kind of takes away the appeal of the idea a bit. Having literally zero-config approach (default to IPFS network) for discovering other libp2p nodes that speak the same set of protocols is quite appealing IMO.

Start by reviewing the DHT provider-record concept and the proposed protocol-derived identifier in this issue. Evaluate protocol filtering, deterministic sorting and hashing, TOTP-based rotation, and DHT participation concerns. Done would require community agreement on the mechanism and a sufficiently concrete specification for implementation.