Launch Jupyter notebook server from Scheduler
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Should we add convenience functionality to co-deploy a notebook server along with the Dask Scheduler?
From a technical perspective, this is orthogonal, and reasonably out of scope. In practice though, this might be convenient for lots of people. We see workflows like this pretty commonly. We even baked it into the Helm chart.
Technically speaking, we could include the Juptyer server in a variety of manners:
1. In the same event loop
2. In a forked process for nice cleanup
3. In a spawned process for resilience
These have different levels of integration, which have different trade offs
More generally, we might think about how we would want to run related processes next to the scheduler, and a general extension point for that. In the simplest possible form, this might just be a preload script or Scheduler Plugin (see https://docs.dask.org/en/latest/setup/custom-startup.html) . I'm curious what else we might want to add on top of that to make this work well.
cc @mmccarty (who asked about this) and @jcrist @ericdill @jacobtomlinson @quasiben who may have thoughts.
Contributor guide
Assessment
This issue has not been assessed yet.