Enable GC of connections without losing ordering constraints

From @edsko on November 7, 2012 9:56

When process A sends a message to process B, we must open a connection from A to B and, currently, keep that connection open for the duration of A's lifetime in order to maintain ordering guarantees. If A sends messages to lots of different processes (think of a server responding to clients) this will result in a space leak.

What we need is a way to garbage collect connections when they are no longer used, but still maintain ordering guarantees. There are (at least) two ways in which we might implement this:

• Client side. When we close the connection, we wait for an acknowledgement from the other side that the connection has been closed. Once we have received this acknowledgement we know that all messages we sent have been received and hence it's safe to open a new connection. If the sender starts sending more messages before receiving the acknowledgement this messages must be buffered.

  We could implement this at the Cloud Haskell level, and have the node controller send the acknowledgement. This introduces the question "what about connections to the node controller itself". We already have implicit reconnect to and from node controllers, implying potential message loss, but what we don't want is reordering of messages to node controllers.

  We could instead implement it at the Network.Transport level and introduce a new OutgoingConnectionClosed event, but there is a technical difficulty here too: outgoing connections don't have identifiers on the sender side, only on the receiver side, so we somehow need to be able identify to the sender which connection got closed, without making connect synchronous. (The sender allocates part of the receiver-side ID, but only half of it.) One easy solution is to introduce

  closeWithAck :: Connection -> Int64 -> IO ()
  

  so that we allow the sender to provide a token which will be returned in the OutgoingConnectionClosed event.

• Server side. When we close the connection, we remember the connection ID (but see discussion above: we don't know connection IDs sender side -- similar solutions could be proposed here). Then when we open a new connection we first send a message saying "all messages you receive on this connection must be delivered after you have got all messages on connection X"; this implies server side buffering.

  Of course, this by itself doesn't gain us much because now we still have to maintain state: the connection ID of the last connection to every process we ever sent a message to. So the question becomes when can we forget this state? This seems to require some sort of acknowledgement from the server, though.

Client side seems the easier way to solve this problem.

Copied from original issue: haskell-distributed/distributed-process#64

No files or tests are named. Start by comparing the client-side and server-side approaches described, then trace the node controller and Network.Transport connection lifecycle. A solution is done when unused connections can be garbage-collected without losing message ordering guarantees, including the acknowledged-close behavior and connection identification concerns.