dask / dask/distributed

asyncio exception when monitoring distributed compute

Open
#8,382 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:
In the example code below, I try to monitor the progress of a cluster compute.
It does return the right results, but there is always an exception as shown below. This exception cannot be suppressed by try/except wrapper I add.

```
(1, 1, 1)
2023-11-30 21:27:09,396 - distributed.scheduler - ERROR -
Traceback (most recent call last):
File "/miniconda3/envs/X/lib/python3.9/site-packages/distributed/utils.py", line 832, in wrapper
return await func(*args, **kwargs)
File "/miniconda3/envs/X/lib/python3.9/site-packages/distributed/scheduler.py", line 7246, in feed
await asyncio.sleep(interval)
File "/miniconda3/envs/X/lib/python3.9/asyncio/tasks.py", line 652, in sleep
return await future
asyncio.exceptions.CancelledError
```

**Minimal Complete Verifiable Example**:

```python
from time import sleep
import dask
from dask.distributed import Client,progress
from dask_jobqueue import SGECluster
from dask import compute, persist, delayed

@dask.delayed
def f():
sleep(1)
return 1

cluster = SGECluster(cores=1, memory='1GB')
cluster.scale(1)
c = Client(cluster)
out=[ f() ]*3

try:
x=persist(*out)
progress(x)
out=compute(x)
except:
print("error")

c.close()
print(out[0])

```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2023.11.0
- Python version: 3.9.18
- Operating System: Red Hat Enterprise Linux release 8.5 (Ootpa)
- Install method (conda, pip, source):
mamba install -c conda-forge dask
mamba install -c conda-forge dask-jobqueue

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.