[inetstack] Delayed ACKs Considered Harmful

There is a well-known undesirable interaction between TCP's "delayed ACK" mechanism and Nagle's Algorithm that can cause unnecessary delays in request-response protocols. (Basically, a sender with a small amount of data to send will wait until all previously sent data is ACK'd, while the receiver will wait to ACK until the timer goes off or it receives a second packet). As a result, many TCP implementations contain a mechanism (usually a socket option) to turn one or both of these off (e.g. TCP_QUICKACK and/or TCP_NODELAY). While the ability to turn off Nagle's Algorithm was the original work-around, Nagle himself argues that delayed ACKs do more harm than his algorithm, and that they should be turned off (but Nagle might be biased :-)).

We currently implement delayed ACKs, but not Nagle. We don't currently implement a mechanism to turn delayed ACKs off.

Since we have fine-grained cooperative scheduling between our stack and the application, we might want to revisit how we implement delayed ACKs. The whole idea of the timeout is to give the application time to respond to the received data, so we can "piggyback" the ACK on that response. But if the application calls wait w/o sending, we're just wasting time if we're waiting for the delayed ACK timer to expire to send the ACK.

Bottom line: We could probably replace the delayed ACK timer with a simple flag (and remove another "watched" variable).

Start by locating the inetstack implementation of delayed ACKs and the associated delayed ACK timer and "watched" variable. Determine how cooperative waits and application responses currently trigger ACKs; done means replacing the timer with a flag while preserving ACK behavior, including when the application waits without sending.