Resource Manager Interface

# Resource Manager Interface

_The first step is hard, but standing up is harder._

Deploying distributed on a remote cluster is a large hurdle between the
curious user and serious adoption. Efforts to improve deployment can
significantly improve community uptake and value.

Unfortunately, there are a large number of deployment solutions so there is no
single piece of code we can write to satisfy most potential users. We have
direct relations with people who employ the following deployment technologies:
1. YARN
2. Mesos
3. SLURM
4. Torque
5. SGE
6. SSH
7. Local processes

Supporting each of these is doable in isolation, but when we consider
supporting all in aggregate we desire a common framework.

Additionally, near-future versions of the distributed scheduler may want to
dynamically request and release worker resources depending on load. Providing
a single resource management interface to the scheduler would allow integration
with several technologies in a sane and maintainable manner.

**Question:** _What does such a resource management interface look like?_
## Example Interface

Let us consider the following signals from scheduler to resource manager (RM)
and back.
- Scheduler -> RM: Please give me more workers
- Scheduler -> RM: I no longer need these workers
- RM -> Scheduler: I plan to take back this worker very soon
- RM -> Scheduler: This worker has unexpectedly died

We could implement this as a Python ResourceManger object that has fuctions for
"Please give me more workers" and "I no longer need these workers" that the
scheduler can call as well as callbacks that the scheduler provides on
ResourceManager creation for "I plan to take back this worker soon" and "this
worker has unexpectedly died".

``` python
class ResourceManager(object):
def __init__(self, on_reclaiming_worker, on_worker_died, **kwargs):
...

def request_workers(self, nworkers, **kwargs):
...

def release_workers(self, worker_ids):
...
```

We might then subclass this interface for processes, SSH, Mesos, Yarn, etc..
## Questions

Different resource management systems employ different information. Yarn
allows the specification of CPUs and Memory and the use of containers, HPC job
schedulers have fixed job durations, Mesos operates a bit differently from all.

The example operations provided above may not fit all resource managers. Some
may provide only a subset of this functionality while some may have a superset.
How do we balance the needs to specialize to a particular system while also
maintaining compatibility with many systems and keeping code simple.

cc @hussainsultan @broxtronix @quasiben @danielfrg