dask / dask/distributed

Error using PBSCluster with multiple nodes

Open
#3,894 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Hi
I'm encountering some strange behaviour when running xarray with a dask client on our PBS Cluster.
When choosing multiple nodes, aka - `cluster.scale()`, the process constantly fails prompting the error below.
The strange thing is that when running only one 1 machine 'cluster.scale(1)' the process runs smoothly
Code to summon workers -

```python
cluster = PBSCluster(queue = 'some_q_name',
project = 'project1',
cores = 16,
memory = '100GB',
processes = 1,
walltime = '48:00:00')
cluster.scale(2)
client = Client(cluster)
```

Error-
```python-traceback

distributed.utils - ERROR - (mean_combine-partial-8d870d263fee8efd863c386e1a41a643, 48, 0, 0, 0)

Traceback (most recent call last):

File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/utils.py", line 656, in log_errors
yield

File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/scheduler.py", line 1736, in add_worker
typename=types[key],

KeyError: "('mean_combine-partial-8d870d263fee8efd863c386e1a41a643', 48, 0, 0, 0)"
distributed.core - ERROR - "('mean_combine-partial-8d870d263fee8efd863c386e1a41a643', 48, 0, 0, 0)"

Traceback (most recent call last):
File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/core.py", line 459, in handle_comm
result = await result
File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/scheduler.py", line 1736, in add_worker
typename=types[key],

KeyError: "('mean_combine-partial-8d870d263fee8efd863c386e1a41a643', 48, 0, 0, 0)"
```

**Environment**:

- Dask version: '2.17.2'
- Python version: 3.7
- Operating System: CentOS Linux release 7.7.1908 (Core)

- Install method (conda, pip, source): conda

Conda list - (newly created env)

```
# Name Version Build Channel

_libgcc_mutex 0.1 main

blas 1.0 mkl

bokeh 2.0.2 py37_0

bzip2 1.0.8 h7b6447c_0

ca-certificates 2020.1.1 0

certifi 2020.4.5.1 py37_0

cftime 1.1.2 py37heb32a55_0

click 7.1.2 py_0

cloudpickle 1.4.1 py_0

curl 7.69.1 hbc83047_0

cytoolz 0.10.1 py37h7b6447c_0

dask 2.17.2 py_0

dask-core 2.17.2 py_0

dask-jobqueue 0.7.0 py_0

distributed 2.17.0 py37_0

freetype 2.9.1 h8a8886c_1

fsspec 0.7.4 py_0

hdf4 4.2.13 h3ca952b_2

hdf5 1.10.4 hb1b8bf9_0

heapdict 1.0.1 py_0

intel-openmp 2020.1 217

jinja2 2.11.2 py_0

jpeg 9b h024ee3a_2

krb5 1.17.1 h173b8e3_0

ld_impl_linux-64 2.33.1 h53a641e_7

libcurl 7.69.1 h20c2e04_0

libedit 3.1.20181209 hc058e9b_0

libffi 3.3 he6710b0_1

libgcc-ng 9.1.0 hdf63c60_0

libgfortran-ng 7.3.0 hdf63c60_0

libnetcdf 4.7.3 hb80b6cc_0

libpng 1.6.37 hbc83047_0

libssh2 1.9.0 h1ba5d50_1

libstdcxx-ng 9.1.0 hdf63c60_0

libtiff 4.1.0 h2733197_1

locket 0.2.0 py37_1

lz4-c 1.9.2 he6710b0_0

markupsafe 1.1.1 py37h7b6447c_0

mkl 2020.1 217

mkl-service 2.3.0 py37he904b0f_0

mkl_fft 1.0.15 py37ha843d7b_0

mkl_random 1.1.1 py37h0573a6f_0

msgpack-python 1.0.0 py37hfd86e86_1

ncurses 6.2 he6710b0_1

netcdf4 1.5.3 py37hbf33ddf_0

numpy 1.18.1 py37h4f9e942_0

numpy-base 1.18.1 py37hde5b4d6_1

olefile 0.46 py37_0

openssl 1.1.1g h7b6447c_0

packaging 20.3 py_0

pandas 1.0.3 py37h0573a6f_0

partd 1.1.0 py_0

pillow 7.1.2 py37hb39fc2d_0

pip 20.0.2 py37_3

psutil 5.7.0 py37h7b6447c_0

pyparsing 2.4.7 py_0

python 3.7.7 hcff3b4d_5

python-dateutil 2.8.1 py_0

pytz 2020.1 py_0

pyyaml 5.3.1 py37h7b6447c_0

readline 8.0 h7b6447c_0

scipy 1.4.1 py37h0b6359f_0

setuptools 47.1.1 py37_0

six 1.15.0 py_0

sortedcontainers 2.1.0 py37_0

sqlite 3.31.1 h62c20be_1

tblib 1.6.0 py_0

tk 8.6.8 hbc83047_0

toolz 0.10.0 py_0

tornado 6.0.4 py37h7b6447c_1

typing_extensions 3.7.4.1 py37_0

wheel 0.34.2 py37_0

xarray 0.15.1 py_0

xz 5.2.5 h7b6447c_0

yaml 0.1.7 had09818_2

zict 2.0.0 py_0

zlib 1.2.11 h7b6447c_3

zstd 1.4.4 h0b5b093_3

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.