dask / dask/distributed

WARNING - Couldn't gather 1 keys, rescheduling even with --contact-address flag

Open
#3,798 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I'm trying to deploy dask on an AWS Fargate cluster, and am starting workers with the following command:
```
dask-worker --preload /app/common.py tcp://dask_scheduler_service.local:8786 --contact-address tcp://dask_worker_service.local:9000 --listen-address tcp://0.0.0.0:9000
```
The `dask_scheduler_service.local` and `dask_worker_service.local` names are from the service discovery service (cloud map).

I'm able to create a client, submit a job and track its progress on the dashboard. When the workers complete all the tasks, I get a
`WARNING - Couldn't gather 1 keys, rescheduling`
message from the client, and this error from the scheduler:
`ERROR - Workers don't have promised key: ['tcp://xxx.xx.xx.xxx:9000'],`
with the IP of the worker (and not `dask_worker_service.local`)

Anything obvious I'm missing?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.