dask / dask/distributed

New users dashboard experience

Open
#6,736 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I just helped a Dask user get the Dashboard running during the SciPy sprints. It was pretty painful and had a bunch of failures along the way that made it hard. I wanted to capture that here in case there are things we can do to improve this experience.

- The user was running Jupyter Lab on a remote machine via SSH, they set up SSH port forwarding to access Jupyter
- They created a Dask cluster with `client = Client()`
- **Failure 1** They clicked the link in the widget that said `http://localhost:8787/status`, which wasn't port forwarded
- **Failure 2** They added `8787` to the port forward list to fix this but some restriction in their org setup meant this didn't work (I think `8888` may have been on an allowlist but `8787` wasn't)
- **Failure 3** While testing they continued to run `client = Client()` which created many clusters and they were confused why the port kept changing
- @jsignell and I got involved at this point to help out. My suggestion was to use [jupyter-server-proxy](https://github.com/jupyterhub/jupyter-server-proxy) so that we could access the dashboard via Jupyter which was known to be working
- **Failure 4** We stopped Jupyter, ran `pip install jupyter-server-proxy` and started Jupyter again. Sadly Jupyter was running from the base environment and we pip installed into the activated environment. So this didn't take effect and took a long time to debug but we eventually got things installed in the right environment (with help from @yuvipanda).
- **Failure 5** Installing `jupyter-server-proxy` upgraded Jupyter which broke the user's Jupyter config and also took a while to debug.
- **Failure 6** Finally we got back into Jupyter, started the Dask cluster and the user clicked the link in the widget again, which was still saying `http://localhost:8787/status` and didn't work
- We then manually navigated to `http://localhost:8888/proxy/8787/status` and things worked 🎉
- Finally we added a config line to fix the widget `dask.config.set({"distributed.dashboard.link": "/proxy/{port}/status"})`

This was an unpleasant experience for a first-time Dask user. It was time-consuming to debug for three maintainers from across Dask and Jupyter.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.