swarm: timeouts

@magik6k is already working on this.

Since Jun 14, 2018.

Types:

1. TCP connect timeout, RTT, expected latency, etc...
2. Per address dial timeouts.
3. Overall time/resources spent on dialing a peer.
4. Default timeout for `DialPeer` and `NewStream`.

Concerns:

1. WAN versus LAN.
2. Overestimate versus expected time.
3. Overriding timeouts.
4. General purpose timeouts (versus special purpose).

Hard requirement: no massive amount of up-front work.

---

# TCP connect timeout

I've implemented this in https://github.com/libp2p/go-tcp-transport/pull/24.

## WAN/LAN

Nothing proposed or implemented.

## Overestimate

We're currently using 5s. That's much longer than a TCP connect should take. Unfortunately, the retry timeout is 3s on linux/windows and 6s on MacOS. So, if we drop a packet, this is an underestimate.

* [ ] Question: Bump this to 10s for MacOS (allow dropping a single packet)?

## Overriding

Due to the fact that we coalesce dials, individual dialers can't override this. However, it can be customized when configuring a libp2p node.

## Generalize

This timeout is entirely defined and specified in go-tcp-transport.

* [ ] Question: Should we generalize this? Later? I'd call it something like `MaxDialRTT` or something like that but I'm not sure where it should go.

Eventually, we could consider getting really fancy and tracking expected RTTs. However, we can do that later.

# Per address dial timeouts

Currently, this timeout lives in the transports package but is (well, will be) enforced by libp2p/go-libp2p-swarm#77.

## WAN/LAN

libp2p/go-libp2p-swarm#70 tackles this in the swarm. IMO, that's the right place to do it.

## Overestimate

We're currently massively overestimating (allowing 60s).

## Overriding

Again, due to the fact that we coalesce dials from multiple callers, it's hard to make it possible to override this. IMO, we should just continue to massively overestimate.

libp2p/go-libp2p-swarm#77 does allow this to be overridden on a per-swarm basis.

## Generalize

This timeout isn't transport specific at all. This could become an issue in the future so we really do need to be careful to make sure we're overestimating here.

# Overall Time

This is difficult to track. Ideally, we'd track time spent tying up file descriptors/CPUs and, once we hit some limit for a given peer, we'd backoff on dialing that peer for a while.

Really, tracking this isn't that hard. The hard part is tracking it in real-time and then stopping when we hit the limit. We'd need the dial limiter to report when it starts working on a dial and when it stops and then we'd need the `dialAddrs` function to keep a running deadline of `budgetRemaining/activeDials` (updated any time `activeDials` changes).

Note: the `activeDials` variable in `dialAddrs` isn't the right one. That doesn't track dials that are actively consuming a file descriptor.

So, this is doable but tricky. IMO, we should do it, but later.

* [ ] Implement dial resource budgeting.

## WAN/LAN

N/A.

## Overestimate

N/A.

## Overriding

Even with dial coalescing, this is fairly straight forward. We'd just add all the time budgets together.

## Generalize

This is already pretty general (it just tracks time). However, it may be too general. Basically, we may want to apply different budgets to different transports based on how much they "cost" (e.g., if they use file descriptors.

* [ ] Question: Should we have multiple budgets? A DialPeerBudget and a DialPeerFDBudget?

# Default timeout for `DialPeer` and `NewStream`.

See https://github.com/libp2p/go-libp2p-net/pull/26 and libp2p/go-libp2p-swarm#77. This timeout exists to limit the amount of time a *user* waits for a connection to be established before giving up (by default).

## WAN/LAN

Not really applicable.

## Overestimate

As this is easy to override, it should be an overestimate. It's currently 1 minute. IMO, most callers to `NewStream` or `DialPeer` won't want to wait nearly that long.

## Overriding

As this applies to each `DialPeer` and/or `NewStream` call, not the underlying dial, it's trivial to override. https://github.com/libp2p/go-libp2p-net/pull/26 specifies the default timeout and provides methods for overriding it in a context.

## Generalize

This is specified in go-libp2p-net so it's already pretty general-purpose.