Lost progress due to worker-client restart
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
I am using ~16 worker_client to generate tasks (read some data files, calculate parameters needed for further processing), and then submit to the same cluster ~200 workers. It works well except cases where the worker process holding the worker_client approaches memory limit, restarted by nanny and then causes all task progress generated by this worker_client to be lost. The scheduler sees this client disconnected, and erase all futures it was pointing to.
**What you expected to happen**:
Actually I am not sure if this is a bug or expected behavior. Is it possible for worker_client processes to have similar resiliency as regular workers? So after a migration by the nanny/scheduler, the new client_worker can connect to all its old futures? or this is not feasible? Thank you.
**Minimal Complete Verifiable Example**:
Sorry can't produce one there.
**Anything else we need to know?**:
**Environment**:
- Dask version: 2.30.1
- Python version: 3.8.5
- Operating System: Ubuntu 20.04.1 LTS
- Install method (conda, pip, source): pip
Contributor guide
Assessment
This issue has not been assessed yet.