Error using PBSCluster with multiple nodes
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Hi
I'm encountering some strange behaviour when running xarray with a dask client on our PBS Cluster.
When choosing multiple nodes, aka - `cluster.scale()`, the process constantly fails prompting the error below.
The strange thing is that when running only one 1 machine 'cluster.scale(1)' the process runs smoothly
Code to summon workers -
```python
cluster = PBSCluster(queue = 'some_q_name',
project = 'project1',
cores = 16,
memory = '100GB',
processes = 1,
walltime = '48:00:00')
cluster.scale(2)
client = Client(cluster)
```
Error-
```python-traceback
distributed.utils - ERROR - (mean_combine-partial-8d870d263fee8efd863c386e1a41a643, 48, 0, 0, 0)
Traceback (most recent call last):
File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/utils.py", line 656, in log_errors
yield
File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/scheduler.py", line 1736, in add_worker
typename=types[key],
KeyError: "('mean_combine-partial-8d870d263fee8efd863c386e1a41a643', 48, 0, 0, 0)"
distributed.core - ERROR - "('mean_combine-partial-8d870d263fee8efd863c386e1a41a643', 48, 0, 0, 0)"
Traceback (most recent call last):
File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/core.py", line 459, in handle_comm
result = await result
File "/work/stavn/anaconda3/envs/Dask_v2/lib/python3.7/site-packages/distributed/scheduler.py", line 1736, in add_worker
typename=types[key],
KeyError: "('mean_combine-partial-8d870d263fee8efd863c386e1a41a643', 48, 0, 0, 0)"
```
**Environment**:
- Dask version: '2.17.2'
- Python version: 3.7
- Operating System: CentOS Linux release 7.7.1908 (Core)
- Install method (conda, pip, source): conda
Conda list - (newly created env)
```
# Name Version Build Channel
_libgcc_mutex 0.1 main
blas 1.0 mkl
bokeh 2.0.2 py37_0
bzip2 1.0.8 h7b6447c_0
ca-certificates 2020.1.1 0
certifi 2020.4.5.1 py37_0
cftime 1.1.2 py37heb32a55_0
click 7.1.2 py_0
cloudpickle 1.4.1 py_0
curl 7.69.1 hbc83047_0
cytoolz 0.10.1 py37h7b6447c_0
dask 2.17.2 py_0
dask-core 2.17.2 py_0
dask-jobqueue 0.7.0 py_0
distributed 2.17.0 py37_0
freetype 2.9.1 h8a8886c_1
fsspec 0.7.4 py_0
hdf4 4.2.13 h3ca952b_2
hdf5 1.10.4 hb1b8bf9_0
heapdict 1.0.1 py_0
intel-openmp 2020.1 217
jinja2 2.11.2 py_0
jpeg 9b h024ee3a_2
krb5 1.17.1 h173b8e3_0
ld_impl_linux-64 2.33.1 h53a641e_7
libcurl 7.69.1 h20c2e04_0
libedit 3.1.20181209 hc058e9b_0
libffi 3.3 he6710b0_1
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libnetcdf 4.7.3 hb80b6cc_0
libpng 1.6.37 hbc83047_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 h2733197_1
locket 0.2.0 py37_1
lz4-c 1.9.2 he6710b0_0
markupsafe 1.1.1 py37h7b6447c_0
mkl 2020.1 217
mkl-service 2.3.0 py37he904b0f_0
mkl_fft 1.0.15 py37ha843d7b_0
mkl_random 1.1.1 py37h0573a6f_0
msgpack-python 1.0.0 py37hfd86e86_1
ncurses 6.2 he6710b0_1
netcdf4 1.5.3 py37hbf33ddf_0
numpy 1.18.1 py37h4f9e942_0
numpy-base 1.18.1 py37hde5b4d6_1
olefile 0.46 py37_0
openssl 1.1.1g h7b6447c_0
packaging 20.3 py_0
pandas 1.0.3 py37h0573a6f_0
partd 1.1.0 py_0
pillow 7.1.2 py37hb39fc2d_0
pip 20.0.2 py37_3
psutil 5.7.0 py37h7b6447c_0
pyparsing 2.4.7 py_0
python 3.7.7 hcff3b4d_5
python-dateutil 2.8.1 py_0
pytz 2020.1 py_0
pyyaml 5.3.1 py37h7b6447c_0
readline 8.0 h7b6447c_0
scipy 1.4.1 py37h0b6359f_0
setuptools 47.1.1 py37_0
six 1.15.0 py_0
sortedcontainers 2.1.0 py37_0
sqlite 3.31.1 h62c20be_1
tblib 1.6.0 py_0
tk 8.6.8 hbc83047_0
toolz 0.10.0 py_0
tornado 6.0.4 py37h7b6447c_1
typing_extensions 3.7.4.1 py37_0
wheel 0.34.2 py37_0
xarray 0.15.1 py_0
xz 5.2.5 h7b6447c_0
yaml 0.1.7 had09818_2
zict 2.0.0 py_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.4 h0b5b093_3
```
Contributor guide
Assessment
This issue has not been assessed yet.