Replication of selected RES capability with scores verification package in CSET
- Dominant language
- Python
- Stars
- 33
- Forks
- 19
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
Issue to capture work underpinning the implementation of scores (https://scores.readthedocs.io/en/stable/) in CSET
We are replicating RES verification capability for research purposes and as backup for MetPlus and will test scores against RES and MetPlus.
I suggest that we list all diagnostics as a tickbox list that get ticket off once done. If you start to work on one of them edit the line and add your user name with @ against it so that we have a clear visual who is working on what and which issue has started. In addition each tickbox will need to become a sub-issue and linked into the gannt chart with start and target date as well as info on parent task if relevant to help us identify blockers. https://github.com/orgs/MetOffice/projects/560/views/2
You can find linked example RES output on the RAL3-LFRic twiki pages (please ask @jwarner8 or @Sylviabohnenstengel if you have difficulty locating them). Essentially we are replicating each of the output diagnostics from the RES runs linked.
**Comparison against surface observations MSLP, T1p5m, RH1p5m, surface wind (10m), total cloud cover**
- [x] include surface obs (@ukmo-huw-lewis PR's)
- [x] RMSE score timeseries [lodged]
- [ ] MAE score timeseries [in progress]
- [ ] ME score timeseries [in progress]
- [ ] timeseries as forecast range (hours)
- [ ] timeseries as forecast range offset by DT (hours)
- [ ] include =/- standard error bars calculated assuming independent observations (check RES code)
- [ ] data equalised and meaned (check res)
**Comparison against surface observations 6 hour ppn accumulation, Surface 1.5m visibility, Surface 10m wind speed, total cloud cover**
- [x] Probability of detection [lodged]
- [x] Equitable threat score [lodged]
- [x] Frequency bias [in progress]
- [x] probability of false detection [in progress]
- [ ] timeseries as forecast range (hours)
- [ ] data equalised and meaned (check res)
**Develop skills scores from scores above (difference between 2 models compared to obs) and apply to above**
See example plots in June quest ppt slid 13
- [ ] check the RES code for calculations
- [ ] RMSE
- [ ] MAE
- [ ] include =/- standard error bars calculated assuming independent observations (check RES code)
- [ ] timeseries as forecast range (hours)
- [ ] timeseries as forecast range offset by DT (hours)
**Develop vertical scores**
- [ ] ME (model - obs) with xaxis forecast range in hours
- [ ] RMSE (model - obs) with x axis forecast range in hours
- [ ] replicate with xaxis forecast range offset by DT
- [ ] equalised and meaned
**Develop probabilistic scores**
- [ ] CRPS
- [ ] RPS
- [ ] Brier Skill Score
- [ ] Rank histograms
- [ ] ROC
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.