Opportunistic Caching

Currently we clean up intermediate results quickly if they are not necessary for any further pending computation. This is good because it minimizes the memory footprint on the workers, often allowing us to process larger-than-distributed-memory computations.

However, this can sometimes be inefficient for interactive workloads when users submit related computations one after the other, so that the scheduler has no opportunity to plan ahead, and instead needs to recompute an intermediate result that was previously computed and garbage collected.

We *could* hold on to some of these results in hopes that the user will request them again. This trades active memory for potential CPU time. Ideally we would hold onto results that:

1. Have a small memory footprint
2. Take a long time to compute
3. Are likely to be requested again (evidenced by recent behavior)

We did this for the single machine scheduler

* http://dask.pydata.org/en/latest/caching.html
* http://matthewrocklin.com/blog/work/2015/08/03/Caching
* https://github.com/dask/cachey

We could do it in the distributed scheduler fairly easily by creating a `SchedulerPlugin` that watched all computations, selected computations to keep based on logic similar to what is currently in [cachey](https://github.com/dask/cachey), and created a fake Client to keep an active reference to those keys in the scheduler.