MetOffice / MetOffice/CSET

Generate synthetic data for tutorials and end to end testing

Open
#1,764 1 comment 1 reaction 1 assignee Claimed by @jwarner8 View on GitHub
enhancement
Dominant language
Python
Stars
33
Forks
19
Avg merge
1d 22h
Merged PRs (30d)
30

Description

If we want to have a complete tutorial that is accessible to colleagues internal and external, we require some sample data to be able to run through CSET to produce all the relevant plots. Additionally, we'd like to do some testing on the workflow and make sure various permutations of options produces sensible results/works successfully.

Issues with sharing model data;
- Hosting on a server takes up a lot of space and needs to be maintained.
- Licencing issues with putting model data on the open web.

I propose creating a standalone script that generates synthetic data in an appropriate pp/nc container (as we see fit) for a range of variables/cycles/ensemble members.

Synthetic data could be created using a numpy array of zeros, apply some mathematical function to the data (like a sin wave, or exponential/gaussian blobs of set frequency), create an iris cube with metadata that matches a model simulation, and then write this to disk.

We could add some time varying function too. For ensembles, we could add some stochastic perturbation that grows in amplitude with lead time (for the purposes of visualisation and testing, noting this is completely synthetic and it doesn't represent the real model system).

This approach means a user could download a script (or even run generate_test_data.py in CSET), point to a location, use minimal libraries (iris, numpy, math), and write data (maybe a few GBs) in an optimal way on their system.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.