Support TCP Fast Open

TcpListener

All of Linux, OS X and Windows are the same, call setsockopt with TCP_FASTOPEN option

// For Linux, value is the queue length of pending packets
int opt = 5;
// For the others, just a boolean value for enable and disable
int opt = 1;

// Call it before listen()
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN, &opt, sizeof(opt));

TcpStream

This is a bit more complicated than TcpListener, because we have to send SYN with the first data packet to make TFO works. APIs are completely different on different platforms:

Linux

Before 4.11, Linux uses MSG_FASTOPEN flag for sendto(), doesn't need to call connect()

// Send SYN with data in buf
ssize_t n = sendto(socket, buf, buf_length, MSG_FASTOPEN, saddr, saddr_len);

After 4.11, Linux provides a TCP_FASTOPEN_CONNECT option

int enable = 1;

// Enable it before connect()
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN_CONNECT, &enable, sizeof(enable));

// Call connect as usual
int ret = connect(socket, saddr, saddr_len);

BSDs (except Darwin)

Uses sendto() as Linux without MSG_FASTOPEN flag

// Set before sendto()
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN, &opt, sizeof(opt));

// Call sendto() without flags
ssize_t n = sendto(socket, buf, buf_length, 0, saddr, saddr_len);

Darwin (OS X)

Darwin supports TFO after OS X 10.11, iOS 9.0, tvOS 9.0 and watchOS 2.0.

It supports with a new syscall connectx.

sa_endpoints_t endpoints;
memset(&endpoints, 0, sizeof(endpoints));
endpoints.sae_dstaddr = &saddr;
endpoints.sae_dstaddrlen = saddr_len;

int ret = connectx(socket,
                   &endpoints,
                   SAE_ASSOCID_ANY,
                   CONNECT_DATA_IDEMPOTENT | CONNECT_RESUME_ON_READ_WRITE,
                   NULL,
                   0,
                   NULL,
                   NULL);

SYN will be sent with the first write or send.

Windows

Windows supports with ConnectEx, since Windows 10. https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nc-mswsock-lpfn_connectex

// TCP_FASTOPEN is required
int ret = setsockopt(socket, IPPROTO_TCP, TCP_FASTOPEN, &opt, sizeof(opt));

// Get LPFN_CONNECTEX
LPFN_CONNECTEX ConnectEx = ...;

// Call it with the first packet of data, SYN will send with this packet's data
LPDWORD bytesSent = 0;
int ret = ConnectEx(socket, saddr, saddr_len, buf, buf_length, &bytesSent, &overlapped);

Challenge

As we can see for those OSes' APIs, (except OS X and Linux > 4.10) connections are made with the first write(), which means that if we want to support TFO, we have to put those code in the first call of std::io::Write::write or tokio::AsyncWrite::poll_write.

There is no other way except customizing a TcpStream from socket() and call different APIs on different OSes while sending the first data packet.

I want to open a discussion here for how in Rust's world to support TFO gracefully.

Related API PRs

1. connectx for OS X: https://github.com/rust-lang/libc/pull/1635
2. TCP_FASTOPEN_CONNECT for Linux: https://github.com/rust-lang/libc/pull/1634
3. TCP_FASTOPEN for Windows: https://github.com/retep998/winapi-rs/issues/856

Review the TcpListener and TcpStream sections first, then compare the platform-specific setsockopt, sendto, connectx, and ConnectEx APIs described in the issue. A complete effort would need a decided cross-platform API and an implementation strategy for sending the first data packet; the issue does not name files or tests to run.