dask / dask/dask-labextension

Better cluster management and discovery

Open
#189 3 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
TypeScript
Stars
329
Forks
67
PR merge metrics
No merged PRs in 30d

Description

## AKA Let's delete dask-labextension's `ClusterManager`

### Goal

For a few years now this extension has had a built-in [cluster manager](https://github.com/dask/dask-labextension/blob/main/dask_labextension/manager.py), which is what powers the user interface on the frontend. The reasons for having this are basically twofold:
1. When the labextension has a handle on the `Cluster` object it is easier to start, stop, and scale those clusters, and build a user interface to do so.
2. If the labextension knows about what clusters are running it is easier to set up proxies for the dashboards.

However, the uptake of the labextension cluster management facility has been limited. Some people use it, but more often I see people create clusters in their notebooks and then use the magnifying glass icon to connect to the dashboard. I think there are several reasons why the cluster manager hasn't been particularly popular:
* Clusters are launched within the Jupyter server process. This can be nice because they can outlive your notebook kernel, but it can also be annoying to have it be tied to your JupyterLab process (killing your server is pretty common, and you may not want it to kill your cluster connections as well).
* Related to the above, even if your clusters outlive restart of the server process, there is no real discovery mechanism to re-connect to them. They might as well be gone, except now they might be costing you money
* Using the dask config to customize cluster creation args is annoying, and changes only take place if you restart your server.
* It doesn't allow you to create more than one type of cluster (e.g., a `KubeCluster` and a `LocalCluster`).

### Proposal

@jacobtomlinson has a new project [`dask-ctl`](https://github.com/dask-contrib/dask-ctl), which has many of the same goals (and similar API) to the `ClusterManager` in this project. However, it also has some benefits that address some of the above problems:
* It has an extensible entrypoints-based system for registering new kinds of clusters with the discovery mechanism
* It can handle multiple cluster types
* Cluster discovery can survive Jupyter server restarts
* I would get to delete some code here :)

Let's explore replacing the `ClusterManager` in this package with `dask-ctl`!

### Possible stumbling blocks

1. `dask-ctl` is still being incubated as a contrib package. How cautious should we be about adopting it as a dependency?
2. There would likely be some features to upstream to that package (e.g., I would want it to also be able to produce a code snippet for connecting to an existing cluster that can be used for insertion into notebooks). Fortunately, I suspect the maintainers would be amenable to such things.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.