dask / dask/distributed

Launch Jupyter notebook server from Scheduler

Open
#3,043 4 comments 0 reactions 0 assignees View on GitHub
discussion
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Should we add convenience functionality to co-deploy a notebook server along with the Dask Scheduler?

From a technical perspective, this is orthogonal, and reasonably out of scope. In practice though, this might be convenient for lots of people. We see workflows like this pretty commonly. We even baked it into the Helm chart.

Technically speaking, we could include the Juptyer server in a variety of manners:

1. In the same event loop
2. In a forked process for nice cleanup
3. In a spawned process for resilience

These have different levels of integration, which have different trade offs

More generally, we might think about how we would want to run related processes next to the scheduler, and a general extension point for that. In the simplest possible form, this might just be a preload script or Scheduler Plugin (see https://docs.dask.org/en/latest/setup/custom-startup.html) . I'm curious what else we might want to add on top of that to make this work well.

cc @mmccarty (who asked about this) and @jcrist @ericdill @jacobtomlinson @quasiben who may have thoughts.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.