Implement REF cataloguing (ingest) for CMIP and development runs
- Dominant language
- Python
- Stars
- 5
- Forks
- 3
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 8
Description
Implement the following REF ingest steps:
1. Ingest standardised runs
2. Ingest CMIP data from manage_cmip for the activities / models / experiments requested by the CMEW user
3. Combine catalogues ready for use in ref solve
(There is probably a technical question about whether these are separate apps or not, which we can defer to planning.)
Dependencies for this issue:
Acceptance criteria:
- Given CMEW configuration specifying at least one standardised model development run and at least one CMIP run
- When new catalogue task (or tasks) run within CMEW
- Then a combined catalogue is generated that is useable by ref solve (could be validated outside of CMEW if solve not yet implemented in that point and/or set up a test in CMEW)
Dependencies:
- We shouldn't start work on the ingest issue(s) until we are confident that we will use the catalogue, so e.g. pending resolution of solve failure (e.g. https://github.com/Climate-REF/climate-ref/issues/494).
- There is no dependency of ingest on configuration files
- REF does require its environment and creates environments for "providers" (ESMValTool, PMP, ...) - investigation ongoing in SSS repo
Contributor guide
Research direction
Start with the CMEW configuration and the new catalogue task or tasks described in the acceptance criteria, then review Climate-REF issue 494 and the ongoing SSS repository investigation. The work is done when a configuration containing at least one standardised development run and one CMIP run produces a combined catalogue usable by ref solve; confirm this with a CMEW test or external validation if solve is not yet implemented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100