Dashboard doesn't show when scheduler does not listen on localhost
- Dominant language
- TypeScript
- Stars
- 329
- Forks
- 67
- PR merge metrics
- No merged PRs in 30d
Description
I am running a JupyterHub deployment that uses JupyterLab and the Dask lab extension to create HTCondorClusters.
The JupyterHub user sessions run inside Docker containers, and so does the Dask scheduler. The Dask scheduler listens on an address of the private Docker network (not on the loopback interface), so that messages from the workers get forwarded from the host address to the Docker address.
I can create an HTCondorCluster normally via the Dask lab extension, but the dashboard buttons do not show up (see image below).

If I look at the requests done by the browser, I see 403 errors when getting e.g. `dask/dashboard/a44f0140-dd9e-4f9b-a5c3-acd32682eb70/individual-plots.json?1652878129159`. The error message is quite descriptive: `Host '172.17.0.2' is not whitelisted. See https://jupyter-server-proxy.readthedocs.io/en/latest/arbitrary-ports-hosts.html for info.` -- where `172.17.0.2` is the private Docker address.
So it seems that the jupyter server proxy extension does not allow to proxy to `172.17.0.2`. At the link suggested by the error message, they recommend to use the option `host_allowlist`, to include in this case `172.17.0.2` in a whitelist.
However, if I include the `c.ServerProxy.host_allowlist`option in the [jupyter notebook config file](https://jupyter-server-proxy.readthedocs.io/en/latest/server-process.html#specifying-config-via-traitlets), it has no effect. The reason is that [DaskDashboardHandler](https://github.com/dask/dask-labextension/blob/main/dask_labextension/dashboardhandler.py#L101) does not load that option and it does not invoke its [ProxyHandler](https://github.com/jupyterhub/jupyter-server-proxy/blob/main/jupyter_server_proxy/handlers.py#L107) superclass constructor with that option as an argument. Therefore, the `172.17.0.2` address is not whitelisted.
Shouldn't the `DaskDashboardHandler` class define a constructor that loads the `host_allowlist` option and passes it on to `ProxyHandler`?
Another thing I was trying was to set `dashboard_address` (in `scheduler_options`, when building the `HTCondorCluster`) to `localhost:port`, so that the dashboard does listen on the loopback interface, differently from the scheduler. That doesn't work either because the Dask lab extension still thinks the dashboard listens on the same address of the scheduler, since [this call](https://github.com/dask/dask-labextension/blob/main/dask_labextension/dashboardhandler.py#L159) is resolved by [this code](https://github.com/dask/distributed/blob/main/distributed/deploy/cluster.py#L327), i.e. the dashboard link is constructed by using the address of the scheduler (which looks like a bug, since the addresses of the scheduler and the dashboard can be different as I explained above).
Any feedback would be appreciated! Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.