Streamline COLIN extract process
- Dominant language
- JavaScript
- Stars
- 23
- Forks
- 62
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
The COLIN extract process has evolved with the needs of the onboarding process. Some of these steps are still manual and we would like to reduce this as much as possible to things more streamlined.
**Tasks**
- [ ] Update postgres extract ddl to include tables derived from base dataset that help with data analysis
- [ ] Update COLIN extract transfer script to include sql to populate derived tables
- [ ] We may need a top level shell script that ties together the different scripts that we used as part of the COLIN extract process. Talk to Argus before starting this work to get a run down of things. [This](https://hackmd.io/@achiu3q/H1X-AK62yl#Retrieve-Latest-CPRD-Postgres-Extract) is a good reference of how the extract is pulled right now for onboarding
- [ ] Look at tying in [restore.sh](https://github.com/bcgov/lear/blob/main/data-tool/restore_extract.sh) script into COLIN extract process
- [ ] Unfreeze corps in extract that are in migration tables
- [ ] Run indexing + VACUUM on extract - skip this by default but user can use a switch to run indexing
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.