dask / dask/dask-labextension

Cannot create a SLURM cluster using dask-labextension

Open
#68 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
329
Forks
67
PR merge metrics
No merged PRs in 30d

Description

Hello,

My name is Clément Buton, I am a Post-doc working at [IPNL](https://www.ipnl.in2p3.fr/?lang=en) on the[ Euclid European space telescope](https://www.euclid-ec.org).

I am currently trying to use dask-labextension with a SLURM cluster on my laboratory grid infrastructure. So far, I have no problem using dask-labextension together with the local cluster configuration or opening a SLURM cluster client "by hand":

```
from dask.distributed import Client
from dask_jobqueue import SLURMCluster
cluster = SLURMCluster(cores=16,
project="euclid_nisp",
local_directory='$TMPDIR',
walltime='01:00:00',
memory="50GB",
queue="normal")
cluster.scale(10)
client = Client(cluster, processes=False)
client
```

My problem starts when I try to configure the dask-labextension (labextension.yaml) with the SLURMCluster as follows:

```
labextension:
factory:
module: 'dask_jobqueue'
class: 'SLURMCluster'
args: []
kwargs: {'cores': 16,
'processes': 1,
'memory': '50GB',
'queue': 'normal',
'walltime': '02:00:00',
'local_directory': '$TMPDIR'}
default:
workers: null
adapt: null
# minimum: 0
# maximum: 10
initial: []
```

I get the following error message when I try to create a new cluster (i.e. click on the +new button): "cannot get address of non-running Server" (cf a copy of my screen [jupyterlab-dask-slurm.pdf](https://github.com/dask/dask-labextension/files/3358676/jupyterlab-dask-slurm.pdf)) which seems to be related to dask.distributed.

I add bellow the versions of the libraries and extensions I am currently using:

```
(py37) ✘-1 ~/dask-slurm
11:09 $ jupyter serverextension list
config dir: /home/euclid/cbuton/.miniconda/envs/py37/etc/jupyter
dask_labextension enabled
- Validating...
dask_labextension 0.4.0a1 OK
jupyter_server_proxy enabled
- Validating...
jupyter_server_proxy OK
jupyterlab enabled
- Validating...
jupyterlab 1.0.1 OK
jupyterlab_slurm enabled
- Validating...
jupyterlab_slurm 0.1.1 OK
nbserverproxy enabled
- Validating...
nbserverproxy OK

(py37) ✔ ~/dask-slurm
11:10 $ conda list
jupyterlab 1.0.1
jupyterlab-server 1.0.0
dask 2.0.0
dask-core 2.0.0
dask-jobqueue 0.5.0
dask-labextension 0.4.0a1
```

Any help would be deeply appreciated.
Thanks in advance, and thanks again on the really nice work you guys are doing,

Sincerely,
Clément

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.