[DISCUSSION] Multiselect & Packet orientation

Discussion of work under #225. Today—October 28, 2019—@raulk, @yusefnapora, @aschmahmann, @Stebalien, and I (with a few others listening in!) had a lengthy discussion about requirements for multiselect and, more broadly, packet oriented libp2p piggy-backing off of our weekly sync. The recording of the meeting can be found here, though it's quite length (begins around 35min). I wanted to open a forum for discussion so that @marten-seemann would have a chance to chime in, and wanted to host it on GitHub so that we can have a public log of the discussion.

In the discussion, I outlined the major differences between the @marten-seemann's in-progress proposal and @Stebalien's original proposal, #95, the gist of which is that Marten's proposal, in its current instantiation, is quite paired down compared to @Stebalien's. This simplification is attractive for reasons pertaining to implementation and correctness, but have drawbacks as well.

Conversation relatively quickly steered towards the role and presence of multiplexing in multiselect's duties. @raulk initially proposed boostrapping into mplex and using mplex to manage, per the QUIC parlence, "flows" of information on given protocols. In this scheme, mplex "flows" would allow for request-response protocols to have responses associated with their requests. @bigs generally pushed back against the idea of requiring the presence of mplex, asserting that it might make sense bake mplex into the multiselect protocol wholesale, leveraging a reserved "flow", e.g. flow 0 a la QUIC, for control messages and messages on protocols that don't care about associating requests with responses.

@raulk wisely pointed out that this wouldn't reduce any of mplex's overhead in that case, and @bigs then realized it may just make sense to use multiselect to agree on a multiplexer, like mplex (which is purportedly packet friendly), in the standard fashion and then use that single multiselect channel to send multiplexed messages. What's interesting about this, is we could still somewhat "bake in" this multiplexer to multistream, making it aware of negotiated protocol IDs. That way you could have flows in packet oriented libp2p while only paying the frame overhead of a single multiselect frame and single mplex frame.

At this point, @raulk pivoted the conversation to talk more generally about the notion of an "embryonic" stream and an upgrade path. @bigs mentioned that @Stebalien's serial-stream seemed like a good candidate for this sort of upgrading (thoughts @Stebalien?). @raulk highlighted the usefulness of a "capabilities" abstraction, in which peers could express more abstract desires (e.g. multiplexing, public key crypto w/ no perfect forward secrecy) and multiselect could facilitate bootstrapping an appropriate connection. This seems to align with the concept of being able to "upgrade" a channel within multiselect to allow for primitive multiplexing, while letting other channels stay "context free" so to speak.

A brief example of the type of flow I was trying to describe earlier with mplex:

A: multiselect/string /my/protocol/1.0 12 // A refers to my protocol as 12
B: multiselect/dynamic /my/protocol/1.0 33 // B refers to it as 33
// now A may send B messages on /my/protocol/1.0 using the dynamic ID 33, and vice versa using the dynamic ID 12
...
A: multiselect/string /mplex/2.0 13 [mplex frame for protocol 33, with some initial request payload]
B: multiselect/dynamic /mplex/2.0 34
// now B may reply within this mplex channel to associate a response with a particular request
B: multiselect/message 13 [mplex frame containing reply to the request on /my/protocol/1.0

I kind of rushed through that, but there are some key ideas:

• Both sender and receiver should probably publish a dynamic ID for a given protocol to prevent redundant round trips.
• Multiselect's packet oriented multiplexing capabilities would be strictly limited to creating one channel per protocol. If you needed additional channels, you'd need to negotiate a packet friendly multiplexer.
• By making this packet friendly multiplexer friendly with multiselect, it could benefit by referencing negotiated dynamic IDs in its frame headers.

The closing conversation focused on the fact that packet oriented libp2p and stream oriented libp2p likely have different requirements and it may make sense to have unique messages within the multiselect family that only apply in one of those situations. I don't think it will be necessary, but it's useful to think about. Furthremore, @raulk highlighted it may be important to have some varint bitset "flag" space in multiselect messages, which could be used to communicate the presence of additional information based on "upgrades" that may have happened earlier. This was the least fleshed out of what we talked about, but seems like a useful concept to leave room for!

Start with issue #225, then read the proposals discussed here, including #95, and review the linked meeting recording from around 35 minutes. The thread is a forum for unresolved requirements around multiselect, packet orientation, multiplexing, and upgrades; done would require an agreed direction and corresponding specification work.