eWaterCycle / eWaterCycle/ewatercycle
Makkink forcing will use excessive memory with default ESMValTool dask.yml
- Dominant language
- Python
- Stars
- 41
- Forks
- 6
- Avg merge
- 8d 23h
- Merged PRs (30d)
- 2
Description
When running the Makkink forcing generation for ERA5 for a range of a couple of years, it can use excessive memory causing the kernel to crash.
However, when defining `~/.esmvaltool/dask.yml` as the following:
```yml
cluster:
type: distributed.LocalCluster
n_workers: 2
threads_per_worker: 1
memory_limit: 5 GiB
```
It runs fine with a memory use <2GB.
The issue probably occurs in the post-processing function where the potential evaporation is computed. However, we do run into conflicting advice:
- Dask [recommends using the distributed scheduler](https://docs.dask.org/en/stable/scheduling.html#local-processes).
- ESMValTool [recommends the local sheduler](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#dask-default-scheduler-configuration) as not all iris (and [ESMF...](https://github.com/JiaweiZhuang/xESMF/issues/71)) functionality works well with Dask Distributed.
Contributor guide
Assessment
This issue has not been assessed yet.