distributed.client - WARNING - Couldn't gather 1 keys & ERROR - Workers don't have promised key

Hello,

I'm running into a problem I cannot seem to find a solution to. I could be doing something wrong... I'm running dask-scheduler on one machine and a worker on another, version 2.5.2 on windows server 2019 off network (w/o internet access). I simplified my code to still generate the error. I've tried changing the memory and number of threads on the worker process which produce the same result. It will run my executable, or calc.exe, but then give me the "distributed.client - WARNING - Couldn't gather 1 keys" & "ERROR - Workers don't have promised key" errors. Any advice would be greatly appreciated!

Thanks for the help!


from dask.distributed import Client, LocalCluster
import os
import subprocess

class dask_parallel():
def __init__(self, function, ip_address='10.12.82.71:8786'):
self.function = function
self.client = Client(address=ip_address)

def dask_map(self, iterable):
result = self.client.map(self.function, iterable)
for i in result:
i.result()
return result

if __name__ == '__main__':
def testing(x):
result = subprocess.run(x)
return result

full_path = "calc.exe"
folder = 'x:/output'
file_names = os.listdir(folder)
cmd_lines = []
test_this = dask_parallel(testing)

for file in file_names:
cmd_lines.append(full_path) # + file)

results = test_this.dask_map(cmd_lines)


distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'testing-cdfc795da591d031697e32f1345ffbbc': ('tcp://10.12.82.59:54174',)}

cmd: dask-scheduler.exe

distributed.scheduler - INFO - -----------------------------------------------

distributed.scheduler - INFO - Local Directory: C:\Users\ADMINI~1\AppData\Local\Temp\2\scheduler-85j285cr

distributed.scheduler - INFO - -----------------------------------------------

distributed.scheduler - INFO - Clear task state

distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: pip install jupyter-server-proxy

distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:8786

distributed.scheduler - INFO - dashboard at: :8787

distributed.scheduler - INFO - Receive client connection: Client-aeb805f8-7aac-11eb-a04c-ecf4bbeaca7d

distributed.core - INFO - Starting established connection

distributed.scheduler - INFO - Register tcp://10.12.82.59:54174

distributed.scheduler - INFO - Starting worker compute stream, tcp://10.12.82.59:54174

distributed.core - INFO - Starting established connection

distributed.scheduler - INFO - Receive client connection: Client-d0802852-7aaf-11eb-a04c-ecf4bbeaca7d

distributed.core - INFO - Starting established connection

distributed.scheduler - INFO - Remove worker tcp://10.12.82.59:54174

distributed.core - INFO - Removing comms to tcp://10.12.82.59:54174

distributed.scheduler - INFO - Lost all workers

distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://10.12.82.59:54174'], testing-cdfc795da591d031697e32f1345ffbbc

NoneType: None

cmd: dask-worker.exe 10.12.82.71:8786 --nthreads 256

distributed.worker - INFO - Start worker at: tcp://10.12.82.59:54174

distributed.worker - INFO - Listening to: tcp://10.12.82.59:54174

distributed.worker - INFO - dashboard at: 10.12.82.59:54175

distributed.worker - INFO - Waiting to connect to: tcp://10.12.82.71:8786

distributed.worker - INFO - -------------------------------------------------

distributed.worker - INFO - Threads: 256

distributed.worker - INFO - Memory: 274.31 GB

distributed.worker - INFO - Local Directory: X:\output\worker-m01x32i6

distributed.worker - INFO - -------------------------------------------------

distributed.worker - INFO - Registered to: tcp://10.12.82.71:8786

distributed.worker - INFO - -------------------------------------------------

distributed.core - INFO - Starting established connection

distributed.worker - INFO - Stopping worker at tcp://10.12.82.59:54174

distributed.nanny - INFO - Worker closed

distributed.nanny - INFO - Closing Nanny at 'tcp://10.12.82.59:54169'

distributed.dask_worker - INFO - End worker