dask / dask/dask-jobqueue

Unable to load scheduler dashboard in SLURMRunner, but can in cluster

Open
#682 1 comment 0 reactions 0 assignees View on GitHub
needs info
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

**Describe the issue**:
Thanks for your time in advance. I have created a simple "hello world" example of a SLURMRunner and SLURMCluster in my environment. I like the interface for the SLURMRunner instead of effectively needing to create wrappers around jobs in the SLURMCluster construct.

I dispatch my SLURMCluster job via sbatch (since my login node cannot run my scheduler) to a worker node (node-01), and then this dispatches additional jobs on my worker nodes (node[01-06]). When I do this, I am able to visit the scheduler dashboard, although I am seeing slightly weird behavior in job allocation (not the point of this post, I need to look into this more).

When I create my SLURMRunner (same as this example https://jobqueue.dask.org/en/stable/runners-overview.html), my jobs are getting allocated and run, but I am unable to load the scheduler dashboard. I get a 404 Page Not Found when I visit the scheduler link output by the client.dashboard_link and also in the scheduler.json file. This is not the same as when the runner spins down, as in this case I get the Connection Refused. Is this expected?

**Minimal Complete Verifiable Example**:
Using the SLURMRunner in my multi-node environment

```python
# Put your MCVE code here
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2023.6.0
- Python version: 3.11.4
- Operating System: CENTOS-7
- Install method: pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.