dask / dask/distributed

Remove `transition_no_worker_memory`

Open
#6,960 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

As mentioned in https://github.com/dask/distributed/pull/6614#issuecomment-1226650397, you can remove `transition_no_worker_memory` and all tests still pass. It's not a logical transition: if a task is waiting to run, it shouldn't mysteriously be able to end up in memory. This is likely a sign of something else going wrong. Arguably `transition_waiting_memory` is similar.

The only valid way we could think of to cause this transition was the special behavior of `Client.scatter` when run on a worker (it allows you to put a key directly into memory). @fjetter's example:
```python
with Scheduler() as s:
with Worker(s) as w:
with Client(s) as c:
c.submit(func, key='foo', ressources={"A": 1})
sleep(1)
c.scatter({"foo": "bar"})
```

The other way it might happen is an unexpected worker completing a task. If we address https://github.com/dask/distributed/issues/6956, and instead ignore those situations, then I'd feel confident removing these strange transitions and only allow tasks to transition to memory from `processing`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.