Review use of multiprocessing in CSET operators
- Dominant language
- Python
- Stars
- 33
- Forks
- 19
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
At least one operator (age of air) using multiprocessing to speed up computation of the age of air. A similar plan to use multiprocessing for extracting transects to speed this up is planned, alongside using this in regridding unstructured data.
The current implementation of CSET is that bake requests a single SPICE allocation, and bakes recipes in parallel, number specified in the site file. There is a risk that we end up using multiprocessing in operator, as well as multiprocessing across recipes. A consequence might be significant slowing across the node and too many threads/processes being executed - ultimately leading to oversubscription relative to the number of cores.
This can lead to quite horrendous scaling due to context switching overhead/memory bandwidth and probably an unhealthy number of I/O calls.
We already scale ntasks with the number of parallel bakes (and number of cpus). This suggests to me there is not much scope for in operator multiprocessing (or we could retain it, and stick to no_cpu=1 to preserve functionality if we change the architecture in future?)
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace the age-of-air operator, bake's parallel recipe execution, and the site-file settings that control parallel bakes and CPUs. Review where multiprocessing is used or planned for transect extraction and unstructured-data regridding, then assess the resulting process and thread counts. Done should be a documented architecture decision and a clear implementation scope that avoids oversubscription.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100