dask / dask/distributed

Lost progress due to worker-client restart

Open
#4,400 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:
I am using ~16 worker_client to generate tasks (read some data files, calculate parameters needed for further processing), and then submit to the same cluster ~200 workers. It works well except cases where the worker process holding the worker_client approaches memory limit, restarted by nanny and then causes all task progress generated by this worker_client to be lost. The scheduler sees this client disconnected, and erase all futures it was pointing to.

**What you expected to happen**:
Actually I am not sure if this is a bug or expected behavior. Is it possible for worker_client processes to have similar resiliency as regular workers? So after a migration by the nanny/scheduler, the new client_worker can connect to all its old futures? or this is not feasible? Thank you.

**Minimal Complete Verifiable Example**:
Sorry can't produce one there.

**Anything else we need to know?**:

**Environment**:

- Dask version: 2.30.1
- Python version: 3.8.5
- Operating System: Ubuntu 20.04.1 LTS
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.