bcgov / bcgov/entity

Streamline COLIN extract process

Open
#30,733 0 comments 0 reactions 0 assignees View on GitHub
Entities - Data Migration
Dominant language
JavaScript
Stars
23
Forks
62
Avg merge
24m
Merged PRs (30d)
1

Description

The COLIN extract process has evolved with the needs of the onboarding process. Some of these steps are still manual and we would like to reduce this as much as possible to things more streamlined.

**Tasks**
- [ ] Update postgres extract ddl to include tables derived from base dataset that help with data analysis
- [ ] Update COLIN extract transfer script to include sql to populate derived tables
- [ ] We may need a top level shell script that ties together the different scripts that we used as part of the COLIN extract process. Talk to Argus before starting this work to get a run down of things. [This](https://hackmd.io/@achiu3q/H1X-AK62yl#Retrieve-Latest-CPRD-Postgres-Extract) is a good reference of how the extract is pulled right now for onboarding
- [ ] Look at tying in [restore.sh](https://github.com/bcgov/lear/blob/main/data-tool/restore_extract.sh) script into COLIN extract process
- [ ] Unfreeze corps in extract that are in migration tables
- [ ] Run indexing + VACUUM on extract - skip this by default but user can use a switch to run indexing

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.