dask / dask/distributed

Cluster component repr hierarchy

Open
#5,216 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

When launching a cluster with a cluster manager like `SSHCluster` there are four different places where worker info exists.

- On the worker system that I am SSHing into there is an instance of `distributed.worker.Worker`.
- On my local system there is an instance of `distributed.deploy.ssh.Worker` which manages the SSH subprocess. This is a subclass of `ProcessInterface` which is being discussed in this PR.
- On the scheduler system there is an instance of `distributed.scheduler.Scheduler` which manages a dictionary of `scheduler_info` that is kept in sync by the worker heartbeat.
- On my local system both the `SSHCluster` and `Client` objects have a copy of the `scheduler_info` dictionary from the `Scheduler` via the RPC.

Today the HTML repr which shows scheduler and worker info is on that `scheduler_info` object. This is because the `scheduler_info` is the simplest way of getting and showing this information to the user.

There are some things to think about here:
- Do the `Worker` and `ProcessInterface` objects have all the same information about the workers that `scheduler_info` does? (I think no)
- Should `distributed.worker.Worker` and `distributed.deploy.ssh.Worker` have reprs that look like the worker dropdowns in `scheduler_info`?
- Should `Cluster` and `Client` reuse the reprs from the `Worker` objects instead of creating its own representation for them?

_Originally posted by @jacobtomlinson in https://github.com/dask/distributed/issues/5181#issuecomment-895879532_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.