dask / dask/distributed

Async disk access: thaw dependencies ahead of execute

Open
#7,643 0 comments 0 reactions 1 assignee Claimed by @crusaderky View on GitHub
memory
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

This is a follow-up to #4424.

One issue with unspilling (asynchronous or not) is that, whenever you start executing a task whose inputs need unspilling, you end up with CPU under-utilization, since you have a task in executing state that's actually busy doing I/O.
When unspilling becomes asynchronous, we gain the option to pre-load the dependencies from disk.

Barring very complex solutions where the worker state machine becomes unspill-aware (that would require a new "unspilling-inputs" task state in between ready and executing, a new asynchronous instruction to match, and a wealth of new transitions), I would like to suggest a simpler, greedy design.

# Proposed design
When a task reaches the top of the ready or constrained heap, but can't transition immediately to executing, the worker state machine fires an `async_get` command (https://github.com/dask/distributed/issues/4424#issuecomment-1464185143) to the SpillBuffer with the list of dependencies. This brings all inputs necessary for the task to the top of the LRU and out of disk.

The output of the command is discarded.
When `Worker.execute` finally runs, it will call `async_get` again, with exactly the same keys. If enough time has passed, all keys are now in fast. If they are still in the middle of unspilling, the SpillBuffer will just return a reference to the already-existing Futures.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.