dask / dask/distributed

RecursionError when using PerformanceReport context manager

Open
#8,578 8 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:

When executing certain custom task graphs with the PerformanceReport context manager I get log warnings like the following:
```
2024-03-13 14:43:10,263 - distributed.sizeof - WARNING - Sizeof calculation failed. Defaulting to -1 B
Traceback (most recent call last):
File ".../site-packages/distributed/sizeof.py", line 17, in safe_sizeof
return sizeof(obj)
File ".../site-packages/dask/utils.py", line 773, in __call__
return meth(arg, *args, **kwargs)
File ".../site-packages/dask/sizeof.py", line 96, in sizeof_python_dict
+ sizeof(list(d.values()))
File ".../site-packages/dask/utils.py", line 773, in __call__
return meth(arg, *args, **kwargs)
File ".../site-packages/dask/sizeof.py", line 59, in sizeof_python_collection
return sys.getsizeof(seq) + sum(map(sizeof, seq))
File ".../site-packages/dask/utils.py", line 773, in __call__
return meth(arg, *args, **kwargs)
```
which repeats until it finally ends with
```
File ".../site-packages/dask/sizeof.py", line 59, in sizeof_python_collection
return sys.getsizeof(seq) + sum(map(sizeof, seq))
RecursionError: maximum recursion depth exceeded
```
The computation still completes correctly and this problem doesn't arise when executing without the performance report.

**Minimal Complete Verifiable Example**:

This is a small example code that reproduces the problem, where I am using the xarray data from https://github.com/pydata/xarray-data/blob/master/rasm.nc.

```python
from dask.distributed import Client, performance_report
import xarray as xr

dask_graph = {"source": (xr.load_dataset, "rasm.nc")}
with Client() as client:
with performance_report(filename="dask-report.html"):
client.get(dask_graph, "source")
```

**Environment**:

- Dask version: 2024.2.0
- Python version: 3.10
- Operating System: Linux
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.