Review functionality: add worker resources on the fly
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
These methods were added in #857 and were not functionally iterated upon since:
https://github.com/dask/distributed/blob/d88c1d236da65f036af6ea05881c84f6b4c9fbb8/distributed/scheduler.py#L6632-L6643
https://github.com/dask/distributed/blob/d88c1d236da65f036af6ea05881c84f6b4c9fbb8/distributed/worker.py#L1755-L1767
They are affected by several problems:
- It seems to be public API (it's not used anywhere internally), but it's undocumented
- It will cause a deadlock if you remove resources below what an enqueued task requires
- It should trigger `_ensure_computing` when increasing resources
- It's prone to race conditions, where the scheduler sends a ComputeTaskEvent for a task but the necessary resources are no longer available
- The matching method in Scheduler is ambiguous: it's called `add_resources`, but its RPC comms handler is called `set_resources`
- The Scheduler handler should use batched comms, not RPC
- `Scheduler.add_resources` is also invoked to replace (not add to) the worker resources at every heartbeat - which is unnecessary
CC @mrocklin (who originally wrote #857), @hendrikmakait
Contributor guide
Assessment
This issue has not been assessed yet.