MetOffice / MetOffice/CSET

Review use of multiprocessing in CSET operators

Open
#2,110 2 comments 0 reactions 0 assignees View on GitHub
small
Dominant language
Python
Stars
33
Forks
19
Avg merge
1d 22h
Merged PRs (30d)
30

Description

At least one operator (age of air) using multiprocessing to speed up computation of the age of air. A similar plan to use multiprocessing for extracting transects to speed this up is planned, alongside using this in regridding unstructured data.

The current implementation of CSET is that bake requests a single SPICE allocation, and bakes recipes in parallel, number specified in the site file. There is a risk that we end up using multiprocessing in operator, as well as multiprocessing across recipes. A consequence might be significant slowing across the node and too many threads/processes being executed - ultimately leading to oversubscription relative to the number of cores.

This can lead to quite horrendous scaling due to context switching overhead/memory bandwidth and probably an unhealthy number of I/O calls.

We already scale ntasks with the number of parallel bakes (and number of cpus). This suggests to me there is not much scope for in operator multiprocessing (or we could retain it, and stick to no_cpu=1 to preserve functionality if we change the architecture in future?)

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace the age-of-air operator, bake's parallel recipe execution, and the site-file settings that control parallel bakes and CPUs. Review where multiprocessing is used or planned for transect extraction and unstructured-data regridding, then assess the resulting process and thread counts. Done should be a documented architecture decision and a clear implementation scope that avoids oversubscription.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.