dask / dask/distributed

Review functionality: add worker resources on the fly

Open
#6,677 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

These methods were added in #857 and were not functionally iterated upon since:
https://github.com/dask/distributed/blob/d88c1d236da65f036af6ea05881c84f6b4c9fbb8/distributed/scheduler.py#L6632-L6643

https://github.com/dask/distributed/blob/d88c1d236da65f036af6ea05881c84f6b4c9fbb8/distributed/worker.py#L1755-L1767

They are affected by several problems:

- It seems to be public API (it's not used anywhere internally), but it's undocumented
- It will cause a deadlock if you remove resources below what an enqueued task requires
- It should trigger `_ensure_computing` when increasing resources
- It's prone to race conditions, where the scheduler sends a ComputeTaskEvent for a task but the necessary resources are no longer available
- The matching method in Scheduler is ambiguous: it's called `add_resources`, but its RPC comms handler is called `set_resources`
- The Scheduler handler should use batched comms, not RPC
- `Scheduler.add_resources` is also invoked to replace (not add to) the worker resources at every heartbeat - which is unnecessary

CC @mrocklin (who originally wrote #857), @hendrikmakait

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.