Per-model preprocessors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 33
- Forks
- 19
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
### What problem does your feature request solve?
Sometimes, in cases like #675, it is useful to apply a transformation to an entire set of model data before any recipes are applied, such as cutting out a region, or fixing something. CSET does not currently have the capability to do that, as modifying a recipe won't allow it to be done per-model.
### Describe the solution you'd like
Add a pre-processing step to the workflow that can optionally run a user defined recipe to prepare the data, per model.
As part of the data time cycling work in #750, I've added the rose config side of a preprocessor system, where transformations can be applied to an entire set of model data before use.
We need to consider how we will define these preprocessors (more recipes?), and how configurable they should be. Do we want YAML recipes, or arbitrary python scripts?
### Describe alternatives you've considered
Bundling this into recipes makes too many separate recipes.
Build logic into the recipes, so it can be done conditionally. While powerful this would vastly increase the complexity of the recipes, making them less reliable and harder to reason about.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the preprocessor configuration work from #750 and the per-model use case in #675. Clarify whether preprocessors should be additional recipes, YAML recipes, or arbitrary Python scripts, and how they fit into the workflow before recipes run. Done means a defined, configurable per-model preprocessing path with an agreed validation strategy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100