Stream Migration Protocol

Libp2p should support protocol agnostic stream migration. This will make upgrading to better transports "seamless".

Requirement

1. Transport agnostic. Really, this means migrating at the stream level.
2. Minimal overhead. Overhead should be at most a small per-stream cost (no additional framing, etc.)
3. No interruption. Reading/writing should be continuous.
4. Transparent. Applications using migratable streams shouldn't notice anything.
5. Correct. There can't be any ambiguity (one side believing the migration happened, the other side disagreeing, etc.).

Protocol

Here we describe a protocol for migrating a "source" stream (stream A) to a "target" stream (stream B).

When opening a stream, if the remote peer supports the stream migration protocol (discovered through identify) and the stream is "long lived" (opt in or opt out?):

1. First, open a stream (stream A).
2. Negotiate the stream migration protocol.
3. Send a message indicating that we're opening a new stream, along with a unique ID for the stream.
4. Negotiate the actual protocol.

Because we "know" that the peer supports this stream migration protocol, we can pipeline these steps so as not to take any additional round-trips.

To migrate a stream:

1. The initiator of the stream migration will open a new stream (stream B), on a the "target" connection (the connection to which we're migrating the stream), to the remote peer. This is the "target" stream for the migration.
2. On stream B, the initiator will negotiate the stream migration protocol.
3. On stream B, the initiator will send a message indicating that we're migrating a stream, along with the ID of the stream we're migrating (the source stream, stream A).
4. The receiver will acknowledge the migration (on stream B) and, from this point onward, treat any "EOF" (close) on stream A as a migration to stream B.
5. When the initiator receives the acknowledgement on stream B, it will close stream A for writing, and start writing on stream B.
6. When the receiver receives the EOF on stream A, it will send an EOF on stream A, switching over to stream B.
7. When the initiator receives an EOF on stream A, it will start reading on stream B.

At this point, the stream is fully migrated.

Resets

If either stream is "reset" before both ends are closed, both streams must be reset and the stream as a whole should be considered "aborted" (reset).

Half-Closed

If stream A was half-closed (either for reading or writing), that state must be replicated on the new stream after the initial handshake. Importantly, there's an edge-case:

1. The receiver tries to stream A for writing.
2. The receiver receives a migration request on stream B, for stream A.
3. The receiver ACKs the migration request.
4. The initiator sees the ACK on stream B.
5. The initiator sees the EOF on stream A, and treats it as the migration EOF.

This is fine. The stream will be migrated and the EOF will be re-played on stream B, leaving stream B in the intended state.

Analysis

• Transport agnostic: This protocol can migrate any stream from any transport to any other stream-based transport. It can even migrate unidirectional and half-closed streams, as long as the new transport supports opening bidirectional streams, and can subsequently half-close them.
• Overhead: This protocol will have a small overhead due to the multistream header, and stream ID, but that shouldn't be much in the grand scheme of things (especially if multistream 2 lands at some point). Importantly, this protocol requires no message framing.
• Interruption: Writing switches instantly to an already prepared stream with no delay.
• Transparent: This protocol supports all the normal stream features (half-close, reset, etc.).
• Correct: There are no "undecidable cases" (to be confirmed in a PoC implementation).