dask / dask/distributed

Distributed scheduler failure case

Open
#5,453 2 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**What happened**:

This is a (very) simplified version of distributed scheduler failure that happens in [CARS](https://github.com/CNES/cars/).
Though the memory needed to compute the task graph is high and the memory resources are limited, there is a task order that guarantees the success of computation, but it is not found by the scheduler.

**Minimal Complete Verifiable Example**:

```python
from typing import List
import dask
from dask_jobqueue import PBSCluster
from dask.distributed import Client
from time import sleep
import numpy as np
```
```python
dask.__version__
```

'2021.09.1'

First, we **start a distributed cluster** with **2 workers**, each with **single thread and 400 Mo of RAM**.

```python
cluster = PBSCluster(project='scheduler-failure', n_workers=2, processes=1, cores=1, memory='400M', local_directory='$TMPDIR', log_directory = '$TMPDIR', walltime='00:30:00', interface='ib0')
```

```python
client = Client(cluster)
```

```python
cluster
```

Tab(children=(HTML(value='

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.