dask / dask/distributed

Reuse worker to scheduler WS connection for scheduling

Open
#8,918 4 comments 0 reactions 0 assignees View on GitHub
needs triage
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

I learned that right now for every task the scheduler creates a new connection that sometimes may not be feasible when the worker is behind the firewall. Since WS communication between client -> scheduler and worker -> scheduler already works, it can be used for scheduling tasks. Although, I understand that there may be some worker to worker communication, it's important to make the system work even though a worker is not directly accessible from the scheduler.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.