dask / dask/distributed

Move disk I/O off of the main event loop thread

Open
#1,276 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently when the worker writes data to disk it does so on the event loop thread. If this data is large this can cause the worker to disappear from the network for large blocks of time.

When we get or put data from `worker.data` that requires communication with disk we probably need to yield on some concurrent.futures task that will happen in a separate thread. I've tried this before and have run into some concurrency issues. This might be a good place for a tornado Lock.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.