OSError: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time

Hi,

I am trying to use dask.distributed on slurm scheduler on a cluster. I tried some very simple example, just trying to make it work. However, the results are not very consistent. I searched online about this issue. Updating tornado doesn't work for me. Here is the code I tried to run on slurm cluster.

The following code basically build a SLURMCluster and run joblib on it so that my parallelization can run over multiple machines. What I do is not using interactive session, but use batch submission. The batch script is attached after the following python code (python 3).
```
"""try_dask_joblib.py"""
import time
from joblib import Parallel, delayed, parallel_backend
import dask
from dask.distributed import Client
from dask_jobqueue import SLURMCluster
import socket

cluster = SLURMCluster(cores=8,
memory="10GB",
project="p30309",
walltime="00:59:59",
queue="short",
job_extra=[
"--output=../std_output/sim_oe_joblib_dask_2_3_2.txt",
"--open-mode=append"],
death_timeout=100)
print(dask.config.get("jobqueue.slurm.death-timeout"))
print(dir(cluster))
print("The worker spec is:\n {}".format(cluster.worker_spec,))
cluster.scale(10) # This line is necessary to start workers.
client = Client(cluster)
print("The worker spec is:\n {}".format(cluster.worker_spec,))
print("The scheduler info is:\n {}".format(client.scheduler_info(),))
print(dir(client))
print(client.status, client.start)

def long_running_function(i):
start_time = time.time()
while time.time()-start_time < 10:
temp_start_time = time.time()
while time.time() - temp_start_time < 1:
i**2
# print(time.time())
print(i, time.time(), time.time()-start_time)
print("The hostname is {}.\n".format(socket.gethostname(),))
return i, time.time(), time.time()-start_time, socket.gethostname()

with parallel_backend('dask', wait_for_workers_timeout=240, n_jobs=40):
res = Parallel(verbose=100, pre_dispatch='4*n_jobs')(
delayed(long_running_function)(i) for i in range(400))

print(res)
```

```
#!/bin/bash
#SBATCH -J cr_model
#SBATCH -A p30309
#SBATCH -p short
#SBATCH -t 0:59:59
#SBATCH -n 2 # Number of all cores for scheduler.
#SBATCH --mem=10000 # Unit is MB.
#SBATCH --output=../std_output/sim_oe_joblib_dask_2_3_2.txt
#SBATCH --mail-type=BEGIN,END,FAIL,REQUEUE

# unload any modules that carried over from your command line session
module purge

# Set your working directory
cd /projects/p30309/CD/CD_github/ #$PBS_O_WORKDIR

# load modules you need to use
module load python/anaconda3.6
source activate py37

# Test joblib with dask
python ./command_script/try_dask_joblib.py
```

__Problem 1__: One problem is that every time, I seems to get different number of workers shown in output `[Parallel(n_jobs=40)]: Using backend DaskDistributedBackend with 56 concurrent workers.` Shouldn't be 80 if everything works fine?

__Problem 2__: I got error when I use python 3.
```
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.131.42:43309'
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.129.110:35846'
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.131.68:43185'
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.131.51:43124'
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.131.68:33089'
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.130.128:39328'
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-j3c7w8s_', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-j3c7w8s_', purging
distributed.diskutils - ERROR - Failed to remove '/projects/p30309/CD/CD_github/worker-j3c7w8s_' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-j3c7w8s_'
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-doc4tad4', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-nml18s3o', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-flged81k', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-mlkny2rl', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-mh78vq49', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-3hxx95rw', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-w5xswvkl', purging
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.129.45:46697'
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.129.45:35281'
distributed.worker - INFO - Start worker at: tcp://172.20.129.110:41685
distributed.worker - INFO - Listening to: tcp://172.20.129.110:41685
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-q441qqyt
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Start worker at: tcp://172.20.131.42:36892
distributed.worker - INFO - Listening to: tcp://172.20.131.42:36892
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-xauyk8g7
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: tcp://172.20.131.68:35567
distributed.worker - INFO - Listening to: tcp://172.20.131.68:35567
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-u31kv4ju
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - Start worker at: tcp://172.20.131.51:34705
distributed.worker - INFO - Listening to: tcp://172.20.131.51:34705
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-b_i71_pe
distributed.worker - INFO - Start worker at: tcp://172.20.131.68:34187
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-q441qqyt', purging
distributed.worker - INFO - Listening to: tcp://172.20.131.68:34187
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-irf7n30m
distributed.worker - INFO - -------------------------------------------------
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-b_i71_pe', purging
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-xauyk8g7', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-irf7n30m', purging
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-u31kv4ju', purging
distributed.worker - INFO - Start worker at: tcp://172.20.130.128:41771
distributed.worker - INFO - Listening to: tcp://172.20.130.128:41771
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-ojdbnzgn
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.130.6:42598'
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-ojdbnzgn', purging
distributed.worker - INFO - Start worker at: tcp://172.20.129.45:44277
distributed.worker - INFO - Listening to: tcp://172.20.129.45:44277
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-n0agt42u
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Start worker at: tcp://172.20.129.45:38788
distributed.worker - INFO - Listening to: tcp://172.20.129.45:38788
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-0xk48yqm
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-n0agt42u', purging
distributed.nanny - INFO - Start Nanny at: 'tcp://172.20.134.54:39716'
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-0xk48yqm', purging
distributed.worker - INFO - Start worker at: tcp://172.20.130.6:36629
distributed.worker - INFO - Listening to: tcp://172.20.130.6:36629
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-yljqfs35
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.diskutils - INFO - Found stale lock file and directory '/projects/p30309/CD/CD_github/worker-yljqfs35', purging
distributed.worker - INFO - Start worker at: tcp://172.20.134.54:33258
distributed.worker - INFO - Listening to: tcp://172.20.134.54:33258
distributed.worker - INFO - Waiting to connect to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 8
distributed.worker - INFO - Memory: 10.00 GB
distributed.worker - INFO - Local Directory: /projects/p30309/CD/CD_github/worker-tm7fx9_4
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Registered to: tcp://172.20.129.23:43394
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
tornado.application - ERROR - Exception in callback functools.partial(>, exception=OSError("Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time")>)
Traceback (most recent call last):
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 221, in connect
_raise(error)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 204, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/kzy816/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/kzy816/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
future.result()
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/worker.py", line 880, in heartbeat
metrics=await self.get_metrics(),
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/core.py", line 735, in send_recv_from_rpc
comm = await self.pool.connect(self.addr)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/core.py", line 869, in connect
connection_args=self.connection_args,
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 230, in connect
_raise(error)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 204, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time
distributed.comm.tcp - WARNING - Closing dangling stream in
tornado.application - ERROR - Exception in callback functools.partial(>, exception=OSError("Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time")>)
Traceback (most recent call last):
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 221, in connect
_raise(error)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 204, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/kzy816/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/kzy816/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
future.result()
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/worker.py", line 880, in heartbeat
metrics=await self.get_metrics(),
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/core.py", line 735, in send_recv_from_rpc
comm = await self.pool.connect(self.addr)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/core.py", line 869, in connect
connection_args=self.connection_args,
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 230, in connect
_raise(error)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 204, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time
tornado.application - ERROR - Exception in callback functools.partial(>, exception=OSError("Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time")>)
Traceback (most recent call last):
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 221, in connect
_raise(error)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 204, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/kzy816/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/kzy816/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
future.result()
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/worker.py", line 880, in heartbeat
metrics=await self.get_metrics(),
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/core.py", line 735, in send_recv_from_rpc
comm = await self.pool.connect(self.addr)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/core.py", line 869, in connect
connection_args=self.connection_args,
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 230, in connect
_raise(error)
File "/home/kzy816/.local/lib/python3.7/site-packages/distributed/comm/core.py", line 204, in _raise
raise IOError(msg)
OSError: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: Timed out trying to connect to 'tcp://172.20.129.23:43394' after 10 s: connect() didn't finish in time
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
distributed.comm.tcp - WARNING - Closing dangling stream in
```

I didn't get useful information by searching over internet.

__Problem 3__:
On python 2, I also get problem that I cannot get enough workers as I specified in the `SLURMCluster()` and using `.scale()` method. Besides, I also got error like following. I tried to change the `--local_directory` Matthew once mentioned in [stackoverflow](https://stackoverflow.com/questions/48657374/what-is-the-default-directory-where-dask-workers-store-results-or-files), but it doesn't solve the problem.

```
distributed.diskutils - ERROR - Failed to remove u'/projects/p30309/CD/CD_github/worker-Ra0Bcx' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-Ra0Bcx'
distributed.diskutils - ERROR - Failed to remove u'/projects/p30309/CD/CD_github/worker-ZVE8Mw' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-ZVE8Mw'
distributed.diskutils - ERROR - Failed to remove u'/projects/p30309/CD/CD_github/worker-ZgTe6A' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-ZgTe6A'
distributed.diskutils - ERROR - Failed to remove u'/projects/p30309/CD/CD_github/worker-HRoPNu' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-HRoPNu'
distributed.diskutils - ERROR - Failed to remove u'/projects/p30309/CD/CD_github/worker-IF84m_' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-IF84m_'
distributed.worker - INFO - Connection to scheduler broken. Reconnecting...
distributed.worker - INFO - Connection to scheduler broken. Reconnecting...
distributed.worker - INFO - Connection to scheduler broken. Reconnecting...
distributed.diskutils - ERROR - Failed to remove u'/projects/p30309/CD/CD_github/worker-IF84m_' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-IF84m_'
distributed.diskutils - ERROR - Failed to remove u'/projects/p30309/CD/CD_github/worker-HRoPNu' (failed in ): [Errno 2] No such file or directory: '/projects/p30309/CD/CD_github/worker-HRoPNu'
```

Comparing with the problem of not getting enough workers, this may be minor issue, even though I can get results.

__Summary :__ I guess dask.distributed is trying to help distributed computation for interactive cases. I am not sure if this is correct way of using dask.distributed for batch submission. I want to be able to use joblib in the client of dask so that my joblib parallelization doesn't have to be constrained in one node of the cluster. I would really appreciate, if anyone can share some best practices and some insights about why the problems are here. Thank you!