Distributed scheduler failure case
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**What happened**:
This is a (very) simplified version of distributed scheduler failure that happens in [CARS](https://github.com/CNES/cars/).
Though the memory needed to compute the task graph is high and the memory resources are limited, there is a task order that guarantees the success of computation, but it is not found by the scheduler.
**Minimal Complete Verifiable Example**:
```python
from typing import List
import dask
from dask_jobqueue import PBSCluster
from dask.distributed import Client
from time import sleep
import numpy as np
```
```python
dask.__version__
```
'2021.09.1'
First, we **start a distributed cluster** with **2 workers**, each with **single thread and 400 Mo of RAM**.
```python
cluster = PBSCluster(project='scheduler-failure', n_workers=2, processes=1, cores=1, memory='400M', local_directory='$TMPDIR', log_directory = '$TMPDIR', walltime='00:30:00', interface='ib0')
```
```python
client = Client(cluster)
```
```python
cluster
```
Tab(children=(HTML(value='
Contributor guide
Assessment
This issue has not been assessed yet.