dask / dask/distributed

Task state "alias" for executing tasks

Open
#7,597 5 comments 0 reactions 0 assignees View on GitHub
diagnostics scheduler
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

There is a common misconception that the scheduler doesn't truly know if a task is executing on the worker or not. While this is certainly not accurate information, it is not entirely true since the worker does submit the tasks that are currently on the thread pool as part of the heartbeat, see

https://github.com/dask/distributed/blob/40456b46598a1b096a580af6ad60c7bb6852a3b1/distributed/worker.py#L1242-L1246

This information is actually mostly unused. The submitted timing information is merely used to update unknown prefix timings, see
https://github.com/dask/distributed/blob/40456b46598a1b096a580af6ad60c7bb6852a3b1/distributed/scheduler.py#L4032-L4039

I'm a bit hesitant of introducing this information as part of scheduling heuristics but in terms of observability this _may_ be interesting and would remove a certain user confusion allowing us disambiguate the state `processing` into `assigned` and `executing` on the dashboard, etc. (The dashboard doesn't really care about heartbeat delays, scheduling does. Hence why I don't really want to incorporate this into scheduling if not truly necessary)

As such, I wonder if we wanted to introduce a no-op/alias state "executing" which would be identical to "processing" for all scheduling matters but may be visualized a bit more explicitly.

Thoughts? @gjoseph92 @crusaderky ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.