MetOffice / MetOffice/CSET

Replication of selected RES capability with scores verification package in CSET

Open
#1,983 3 comments 0 reactions 7 assignees Claimed by @ukmo-huw-lewis View on GitHub
super
Dominant language
Python
Stars
33
Forks
19
Avg merge
1d 22h
Merged PRs (30d)
30

Description

Issue to capture work underpinning the implementation of scores (https://scores.readthedocs.io/en/stable/) in CSET

We are replicating RES verification capability for research purposes and as backup for MetPlus and will test scores against RES and MetPlus.

I suggest that we list all diagnostics as a tickbox list that get ticket off once done. If you start to work on one of them edit the line and add your user name with @ against it so that we have a clear visual who is working on what and which issue has started. In addition each tickbox will need to become a sub-issue and linked into the gannt chart with start and target date as well as info on parent task if relevant to help us identify blockers. https://github.com/orgs/MetOffice/projects/560/views/2

You can find linked example RES output on the RAL3-LFRic twiki pages (please ask @jwarner8 or @Sylviabohnenstengel if you have difficulty locating them). Essentially we are replicating each of the output diagnostics from the RES runs linked.

**Comparison against surface observations MSLP, T1p5m, RH1p5m, surface wind (10m), total cloud cover**
- [x] include surface obs (@ukmo-huw-lewis PR's)
- [x] RMSE score timeseries [lodged]
- [ ] MAE score timeseries [in progress]
- [ ] ME score timeseries [in progress]
- [ ] timeseries as forecast range (hours)
- [ ] timeseries as forecast range offset by DT (hours)
- [ ] include =/- standard error bars calculated assuming independent observations (check RES code)
- [ ] data equalised and meaned (check res)

**Comparison against surface observations 6 hour ppn accumulation, Surface 1.5m visibility, Surface 10m wind speed, total cloud cover**
- [x] Probability of detection [lodged]
- [x] Equitable threat score [lodged]
- [x] Frequency bias [in progress]
- [x] probability of false detection [in progress]
- [ ] timeseries as forecast range (hours)
- [ ] data equalised and meaned (check res)

**Develop skills scores from scores above (difference between 2 models compared to obs) and apply to above**
See example plots in June quest ppt slid 13
- [ ] check the RES code for calculations
- [ ] RMSE
- [ ] MAE
- [ ] include =/- standard error bars calculated assuming independent observations (check RES code)
- [ ] timeseries as forecast range (hours)
- [ ] timeseries as forecast range offset by DT (hours)

Image

**Develop vertical scores**
- [ ] ME (model - obs) with xaxis forecast range in hours
- [ ] RMSE (model - obs) with x axis forecast range in hours
- [ ] replicate with xaxis forecast range offset by DT
- [ ] equalised and meaned

**Develop probabilistic scores**
- [ ] CRPS
- [ ] RPS
- [ ] Brier Skill Score
- [ ] Rank histograms
- [ ] ROC

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.