Generate synthetic data for tutorials and end to end testing
- Dominant language
- Python
- Stars
- 33
- Forks
- 19
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
If we want to have a complete tutorial that is accessible to colleagues internal and external, we require some sample data to be able to run through CSET to produce all the relevant plots. Additionally, we'd like to do some testing on the workflow and make sure various permutations of options produces sensible results/works successfully.
Issues with sharing model data;
- Hosting on a server takes up a lot of space and needs to be maintained.
- Licencing issues with putting model data on the open web.
I propose creating a standalone script that generates synthetic data in an appropriate pp/nc container (as we see fit) for a range of variables/cycles/ensemble members.
Synthetic data could be created using a numpy array of zeros, apply some mathematical function to the data (like a sin wave, or exponential/gaussian blobs of set frequency), create an iris cube with metadata that matches a model simulation, and then write this to disk.
We could add some time varying function too. For ensembles, we could add some stochastic perturbation that grows in amplitude with lead time (for the purposes of visualisation and testing, noting this is completely synthetic and it doesn't represent the real model system).
This approach means a user could download a script (or even run generate_test_data.py in CSET), point to a location, use minimal libraries (iris, numpy, math), and write data (maybe a few GBs) in an optimal way on their system.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.