Move disk I/O off of the main event loop thread
Open
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Currently when the worker writes data to disk it does so on the event loop thread. If this data is large this can cause the worker to disappear from the network for large blocks of time.
When we get or put data from `worker.data` that requires communication with disk we probably need to yield on some concurrent.futures task that will happen in a separate thread. I've tried this before and have run into some concurrency issues. This might be a good place for a tornado Lock.
Contributor guide
Assessment
This issue has not been assessed yet.