bcgov / bcgov/CCISS

Set up GitHub Actions to automate output QC/QA & server updates

Open
#48 0 comments 0 reactions 0 assignees View on GitHub
Data management
Dominant language
R
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Several CCISS workflow (and subworkflows) outputs need to be checked for consistency with one another and then uploaded to different cloud servers.
Bellow is a "QC workflow" that needs to be triggered everytime one of the objects with checkmarks is changed:

![Image](https://github.com/bcgov/CCISS/assets/22214224/432524c5-a218-4aa6-8153-bd0e51af5e90)

Once the QC workflow completes successfully, it should trigger a second workflow that uploads a target object (not necessarily all the objects with checkmarks above) to cloud server ("uploading workflow"). Bellow is an example of an uploading workflow with feasibility tables:

![Image](https://github.com/bcgov/CCISS/assets/22214224/9a2adff9-2a03-4c9c-96d3-5308c9f16769)

Notes:
- Both the QC workflow and uploading workflow are implemented as GHActions.
- The QC workflow needs to happen across several GH repos (any repo that alters a checkmarked object).
- The QC will be implemented as an FFEC action (because it's essentially the same across many GH repos) - an FFEC actions repository will need to be created
- The uploading workflow will happen in repos that _host data_ (e.g. bcgov/SS_feasTable repo).*
- QC workflow will need to download objects to be checked from where they were uploaded to - objects cannot be large.

\* workflows that produce large datasets (e.g. BGC projections) automatically upload them to cloud servers, no GHActions needed.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no concrete files, so begin by inspecting the example uploading workflow in bcgov/SS_feasTable and mapping the repositories that alter the checkmarked objects. Trace where those objects are uploaded and how they can be downloaded for QC. Done means a reusable QC workflow runs across the relevant repositories and a successful QC run triggers the intended server upload.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.