MetOffice / MetOffice/CSET

Per-model preprocessors

Open
#807 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
33
Forks
19
Avg merge
1d 22h
Merged PRs (30d)
30

Description

### What problem does your feature request solve?

Sometimes, in cases like #675, it is useful to apply a transformation to an entire set of model data before any recipes are applied, such as cutting out a region, or fixing something. CSET does not currently have the capability to do that, as modifying a recipe won't allow it to be done per-model.

### Describe the solution you'd like

Add a pre-processing step to the workflow that can optionally run a user defined recipe to prepare the data, per model.

As part of the data time cycling work in #750, I've added the rose config side of a preprocessor system, where transformations can be applied to an entire set of model data before use.

We need to consider how we will define these preprocessors (more recipes?), and how configurable they should be. Do we want YAML recipes, or arbitrary python scripts?

### Describe alternatives you've considered

Bundling this into recipes makes too many separate recipes.

Build logic into the recipes, so it can be done conditionally. While powerful this would vastly increase the complexity of the recipes, making them less reliable and harder to reason about.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the preprocessor configuration work from #750 and the per-model use case in #675. Clarify whether preprocessors should be additional recipes, YAML recipes, or arbitrary Python scripts, and how they fit into the workflow before recipes run. Done means a defined, configurable per-model preprocessing path with an agreed validation strategy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.