Allow fixed fields to be processed and archived alongside other data
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 23
Description
Migrated Jira Issue CDDSO-92
Jira creation date 25/04/2022 17:25
-------
At present the inclusion of fixed fields in a round of processing is a very manual process. It would be useful to consider the changes needed to enable this. My run through from arise has the following features;
Extract -- successfully did nothing
Configure – had to add an ancilliary file to enable fx
Convert – not designed to allow single cycle ancilliaries, would need “ancil” stream to be added
QC – no time axis found in files so failed
Store – no time bounds in file names so failed.
Historically we haven’t bothered producing the 9 data sets for anything other than the CMIP6 piControl runs which I did by hand. It would be useful to work out how to adapt to allow for this
-------
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how fixed fields and ancillary streams move through Extract, Configure, Convert, QC, and Store, using the CMIP6 piControl workflow as the comparison case. Determine what changes are needed for fixed fields to pass each stage and be archived with other data; done means the full processing path works without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100