Support for TCP fast open / TLS early data

TCP has a feature now where you can send some data along with the initial SYN packet, called TCP fast open (LWN article, rfc). (At least, if you've made a connection to the same server somewhat recently.) Linux has actually supported this for quite some time (some support since 3.7, complete support since 3.16). This is a bit tricky because the point of the 3-way handshake is to establish a reliable connection, and this lets you send data before that, so it's intrinsically unreliable. In particular a server might randomly seem to get multiple connections and read this data multiple times, even though the client only made one connection, so you can only use this as an opt-in thing with protocols where you know the initial data being sent is idempotent.

TLS 1.3 has a semantically very similar feature, where you can send data along with the initial handshake, called "early data". It has the same problem with requiring idempotency (and the same requirement that you have made a connection to the server before), so it has to be opt-in. It doesn't actually exist as something one can use yet though. (Needs openssl 1.1.1, plus support in wrappers.)

We should support these in our high-level API on both the client and server side. I think that on the client side, this would mean something like an extra early_data= argument to open_tcp_stream or open_ssl_over_tcp_stream. And on the server side, something like an allow_early_data argument to open_tcp_listeners/open_ssl_over_tcp_listeners (defaulting to False)? I guess for open_tcp_listeners this should be a "queue length" (to avoid DoS issues).

For TLS early open, we'll also want to expose some of the richer API that openssl exposes on the SSLStream API. (Or maybe this is the only thing we want to do on the server side?) For TCP fast open, AFAICT the socket API doesn't give any way to distinguish between regular data and fast-open data, so this is unnecessary.

Also, I believe that in general the initial part of a TLS client handshake is always idempotent, so open_ssl_over_tcp_stream should use TCP fast open by default, and likewise for open_ssl_over_tcp_listeners.

However, there is a subtlety when TLS early data is combined with TCP fast open: there is a rule that servers are supposed to reject re-use of TLS early data tokens, so (a) on the server side maybe we need to help implement that somehow?, (b) on the client side, we can't necessarily make one SSLObject and then pass its data to open_tcp_stream, because if happy-eyeballs kicks in then we'll end up sending that same data multiple times. OTOH maybe this is fine because (a) happy eyeballs rarely kicks in, and (b) when it does whichever connection wins is usually going to be the one whose SYN was received first, and (c) even if we do end up using a connection where the token was invalidated by another happy-eyeballs connection, TLS transparently falls back on a regular handshake. I think.

Testing this is going to be super annoying. I'm not sure how to even do it except by like, shelling out to tcpdump or something.

Cross-platform support will also be a bit tricky. On Windows you have to set TCP_FASTOPEN and then use ConnectEx, which means you MUST use IOCP. On MacOS, you have to use the odd connectx, which Python doesn't currently wrap.

This page has some useful notes: https://dnsprivacy.org/wiki/display/DP/TCP+Fast+Open#TCPFastOpen-OSX

Start by reading the open_tcp_stream, open_ssl_over_tcp_stream, open_tcp_listeners, and open_ssl_over_tcp_listeners APIs, then review the Linux, Windows, and macOS constraints described in the issue. Check the OpenSSL SSL_read_early_data interface and investigate tcpdump-based testing; done means the client and server APIs support the requested TCP fast open and TLS early-data behavior with suitable tests and cross-platform handling.