Discussion: Adopting a pattern/library for managing long lived go-routines

# The issue

In many parts of go-IPFS, we launch go routines that run a long time -- essentially for as long as the ipfs daemon is running, or for the lifespan of a command or session. These are usually run-loops that have a structure like this:

```go
for {
select {
case someData <- someChannel:
doSomething(someData)
case ...
case <- doneIndicator:
return
}
}
```

We need to track these go-routines enough to make sure they quit somehow -- otherwise we're leaking memory. We might also need to track them so we can restart them if something goes wrong.

Appeal to authority:
https://dave.cheney.net/2016/12/22/never-start-a-goroutine-without-knowing-how-it-will-stop
https://rakyll.org/leakingctx/

# Prior Art

1. OS Processes - The operating system as a notion of spawning, tracking, and closing long live routines -- they're called processes!
2. Erlang/OTP - Other concurrency oriented languages are much more explicit in calling out these long lived routines and providing mechanisms for spawning and managing them. Erlang maintains supervision trees -- essentially workers who do work (usually a GenServer or other Behavior) and supervisors who track, close, and restart these workers

# Possible Solutions

### Contexts -- a.k.a -- What we're (mostly) doing now

The most common pattern here in the existing code is to just use contexts, either for the lifespan of the daemon or the lifespan of an incoming API requests.

Benefits:
1. Contexts are standard
2. We're already using them

Downsides:
1. Context api is a bit weird in terms of semantics for the goal here. If I want to start a go routine and have ability to shut it down I'd do something like this:

startup childRoutine

```go
childCtx, childCancel = context.WithCancel(parentCtx)
go childRoutine(childCtx)
```

shutDown childRoutine

```go
childCancel()
```

childRoutine:

```
func childRoutine(ctx context.Context) {
for {
select {
...
case <-ctx.Done():
}
}
}
```

It works, it's just a bit clunky:
1. How do I know what my cancel function will actually cancel? Especially if I want to pass it around. A cancel function it a pretty weird way to refer to a goroutine I might want to kill.
2. I have to rely on my child routine to do the right thing and listen to the cancel. No SIGKILL here :)
3. Cancel is more of a sigterm without a wait for it to be handled -- after I call cancel, my child routine will shutdown IN THE FUTURE.
4. If there are hierarchies of routines, they are very implicit.

One possible solution would simply be to call out that we're already doing this and establish a best practices doc of some sort to avoid some of the clunky inconsistencies in the code

### GoProcess

GoProcess is a lightweight library written by our fearless leader @jbenet and used sporadically in parts of the project. It's primarily inspired by the OS model for spawning and managing processes.

Benefits:
1. Syntax is cleaner that context, without adding a whole lot more
2. It's already in use in parts of go-bitswap
3. Pretty lightweight

Downsides:
1. It's just able to spawn processing and shut them down in groups. Lacks some of the more classic patterns of supervision of the Erlang/OTP model
2. Not widely adopted - we'd be using a non-standard library and Juan is likely too busy to maintain it so we'd be doing that ourselves

### go-sup

Go-sup implements supervision trees for Go and is created by our own @warpfork . In comparison to go-process, it provides more control over starting and managing processes, and provides some mechanism for tracking errors, as well as some basic behaviors for each task.

Benefits:
1. More control and tracking of processes
2. Handles errors more explicitly, and could be extended to handle restarts
3. Uses idioms from Erlang/OTP, which was the gold-standard for concurrent programming at least till Go and Rust showed up :)
4. Unlike Juan, @warpfork is actively contributing to the team and might be able to be more responsive.

Downsides:
1. Similarly non-standard like goprocess and not widely adopted. Under the hood, it also uses other libraries @warpfork wrote.
2. Syntax and concepts are more heavyweight, requires more spinning-up on how it works.

Maybe @warpfork can elaborate in the comments on his library

### Use Widely Adopted Library

There are a couple of more widely adopted libraries available for doing supervision:
[Suture](https://github.com/thejerf/suture) - Implementation of supervision trees for go w/ 800 stars, stability, blog post and docs
[ProtoActor](https://github.com/AsynkronIT/protoactor-go) - Full implementation of the Actor model for concurrency for go -- has 2400 stars, created by the person who wrote Akka.net (Well known actor model library)

Benefits:
1. Maturity & Adoption
2. Don't have to handle development

Downsides:
1. Third Party Dependency not maintained by us and all that entails

Suture in particular looks relatively lightweight and reasonable.

Discuss!