dask / dask/distributed

Move time-consuming activities off of the event loop thread

Open
#1,214 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Currently operations like serialization/deserialization, disk I/O, and compression all take place on the event loop thread. This is convenient because it avoids complex issues that can arise from concurrency. However when users pass around large pieces of data this can block the event loop for a non-trivial period which can damage the smoothness of inter-node communication.

So in some cases we might want to move these processes off-thread using a `concurrent.futures.ThreadPool`. I did this a while ago here: https://github.com/dask/distributed/pull/947 . It might be time to reconsider and expand that work.

cc @pitrou

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.